[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:03:28 dist-test-slave-69rg krb5kdc[6397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:03:28 dist-test-slave-69rg krb5kdc[6397](info): set up 2 sockets
Aug 12 07:03:28 dist-test-slave-69rg krb5kdc[6397](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260812 07:03:29.951267 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.176s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:03:30 dist-test-slave-69rg krb5kdc[6397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:03:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:03:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
I20260812 07:03:30.155386 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:46259
--webserver_interface=127.6.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:36761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:46259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:03:30.484503 6411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:03:30.484812 6411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:03:30.484905 6411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:03:30.492172 6411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:03:30.492302 6411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:03:30.492352 6411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:03:30.492410 6411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:03:30.492456 6411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:03:30.502349 6411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:36761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:46259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:46259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:03:30.503685 6411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:03:30.505080 6411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:03:30.517892 6420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:30.517872 6418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:30.518608 6417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:03:30.518896 6411 server_base.cc:1034] running on GCE node
I20260812 07:03:30.519779 6411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:03:30.521242 6411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:03:30.522514 6411 hybrid_clock.cc:648] HybridClock initialized: now 1786518210522427 us; error 68 us; skew 500 ppm
I20260812 07:03:30.522902 6411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:03:30 dist-test-slave-69rg krb5kdc[6397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518210, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:03:30.526121 6411 init.cc:377] Logged in from keytab as kudu/127.6.61.62@KRBTEST.COM (short username kudu)
I20260812 07:03:30.527729 6411 webserver.cc:460] Webserver started at http://127.6.61.62:39439/ using document root <none> and password file <none>
I20260812 07:03:30.528434 6411 fs_manager.cc:359] Metadata directory not provided
I20260812 07:03:30.528584 6411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:03:30.528908 6411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:03:30.532187 6411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "7b1515f1dbf547a6aa54c21578d7ac5e"
format_stamp: "Formatted at 2026-08-12 07:03:30 on dist-test-slave-69rg"
server_key: "2e53606cec7591631b44461cb4e05797"
server_key_iv: "590e71d72a89b07dbf3ea03ac2744626"
server_key_version: "encryptionkey@0"
I20260812 07:03:30.532985 6411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "7b1515f1dbf547a6aa54c21578d7ac5e"
format_stamp: "Formatted at 2026-08-12 07:03:30 on dist-test-slave-69rg"
server_key: "2e53606cec7591631b44461cb4e05797"
server_key_iv: "590e71d72a89b07dbf3ea03ac2744626"
server_key_version: "encryptionkey@0"
I20260812 07:03:30.538357 6411 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260812 07:03:30.541962 6429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:30.543251 6411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:03:30.543442 6411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "7b1515f1dbf547a6aa54c21578d7ac5e"
format_stamp: "Formatted at 2026-08-12 07:03:30 on dist-test-slave-69rg"
server_key: "2e53606cec7591631b44461cb4e05797"
server_key_iv: "590e71d72a89b07dbf3ea03ac2744626"
server_key_version: "encryptionkey@0"
I20260812 07:03:30.543638 6411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:03:30.562357 6411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:03:30.567050 6411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:03:30.567404 6411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:03:30.589066 6411 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:46259
I20260812 07:03:30.589169 6489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:46259 every 8 connection(s)
I20260812 07:03:30.590692 6411 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:03:30.595163 6490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:03:30.595871 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 6411
I20260812 07:03:30.596244 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:03:30.599104 6388 external_mini_cluster.cc:1428] Setting key 04794a46c65fbb49316e6c369eca7dbd
I20260812 07:03:30.609164 6490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b1515f1dbf547a6aa54c21578d7ac5e: Bootstrap starting.
I20260812 07:03:30.612146 6490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b1515f1dbf547a6aa54c21578d7ac5e: Neither blocks nor log segments found. Creating new log.
I20260812 07:03:30.613143 6490 log.cc:824] T 00000000000000000000000000000000 P 7b1515f1dbf547a6aa54c21578d7ac5e: Log is configured to *not* fsync() on all Append() calls
I20260812 07:03:30.616597 6490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b1515f1dbf547a6aa54c21578d7ac5e: No bootstrap required, opened a new log
I20260812 07:03:30.625591 6490 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b1515f1dbf547a6aa54c21578d7ac5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1515f1dbf547a6aa54c21578d7ac5e" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 46259 } }
I20260812 07:03:30.626044 6490 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b1515f1dbf547a6aa54c21578d7ac5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:03:30.626168 6490 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b1515f1dbf547a6aa54c21578d7ac5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b1515f1dbf547a6aa54c21578d7ac5e, State: Initialized, Role: FOLLOWER
I20260812 07:03:30.626819 6490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b1515f1dbf547a6aa54c21578d7ac5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1515f1dbf547a6aa54c21578d7ac5e" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 46259 } }
I20260812 07:03:30.627121 6490 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b1515f1dbf547a6aa54c21578d7ac5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:03:30.627274 6490 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b1515f1dbf547a6aa54c21578d7ac5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:03:30.627456 6490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b1515f1dbf547a6aa54c21578d7ac5e [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:03:30.629717 6490 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b1515f1dbf547a6aa54c21578d7ac5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1515f1dbf547a6aa54c21578d7ac5e" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 46259 } }
Aug 12 07:03:30 dist-test-slave-69rg krb5kdc[6397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:03:30.630406 6490 leader_election.cc:302] T 00000000000000000000000000000000 P 7b1515f1dbf547a6aa54c21578d7ac5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b1515f1dbf547a6aa54c21578d7ac5e; no voters:
I20260812 07:03:30.631150 6490 leader_election.cc:288] T 00000000000000000000000000000000 P 7b1515f1dbf547a6aa54c21578d7ac5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:03:30.631283 6495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b1515f1dbf547a6aa54c21578d7ac5e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:03:30.632699 6495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b1515f1dbf547a6aa54c21578d7ac5e [term 1 LEADER]: Becoming Leader. State: Replica: 7b1515f1dbf547a6aa54c21578d7ac5e, State: Running, Role: LEADER
I20260812 07:03:30.633324 6495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b1515f1dbf547a6aa54c21578d7ac5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1515f1dbf547a6aa54c21578d7ac5e" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 46259 } }
I20260812 07:03:30.633620 6490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b1515f1dbf547a6aa54c21578d7ac5e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:03:30.636664 6497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b1515f1dbf547a6aa54c21578d7ac5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b1515f1dbf547a6aa54c21578d7ac5e. Latest consensus state: current_term: 1 leader_uuid: "7b1515f1dbf547a6aa54c21578d7ac5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1515f1dbf547a6aa54c21578d7ac5e" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 46259 } } }
I20260812 07:03:30.636580 6496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b1515f1dbf547a6aa54c21578d7ac5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b1515f1dbf547a6aa54c21578d7ac5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1515f1dbf547a6aa54c21578d7ac5e" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 46259 } } }
I20260812 07:03:30.636993 6497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b1515f1dbf547a6aa54c21578d7ac5e [sys.catalog]: This master's current role is: LEADER
I20260812 07:03:30.636992 6496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b1515f1dbf547a6aa54c21578d7ac5e [sys.catalog]: This master's current role is: LEADER
I20260812 07:03:30.648056 6492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:30.604251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48528 (local address 127.6.61.62:46259)
0812 07:03:30.605251 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:30.605270 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:30.606819 (+ 1549us) server_negotiation.cc:407] Connection header received
0812 07:03:30.616258 (+ 9439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:30.616284 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:30.616744 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:30.617179 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:30.619959 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:30.622194 (+ 2235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:30.623990 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:30.625077 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:30.625277 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:30.632315 (+ 7038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:30.632329 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:30.632349 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:30.632400 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:30.637995 (+ 5595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:30.639492 (+ 1497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:30.639505 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:30.639517 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:30.639784 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:30.640580 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:30.640588 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:30.640594 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:30.641544 (+ 950us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:30.645348 (+ 3804us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:30.645676 (+ 328us) server_negotiation.cc:299] Negotiation successful
0812 07:03:30.646229 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":441,"threads_started":1}
I20260812 07:03:30.649523 6502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:03:30.656361 6502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:03:30.667246 6502 catalog_manager.cc:1347] Generated new cluster ID: 97b92c83935244db88492b211b741240
I20260812 07:03:30.667376 6502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:03:30.675884 6502 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:03:30.677690 6502 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:03:30.683056 6502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b1515f1dbf547a6aa54c21578d7ac5e: Generated new TSK 0
I20260812 07:03:30.683867 6502 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
I20260812 07:03:30.746711 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.1:0
--local_ip_for_outbound_sockets=127.6.61.1
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:46259
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:36761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:03:31.091589 6517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:03:31.091897 6517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:03:31.092067 6517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:03:31.099437 6517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:03:31.099555 6517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:03:31.099736 6517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.1
I20260812 07:03:31.109756 6517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:36761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:46259
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.1
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:03:31.111205 6517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:03:31.112604 6517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:03:31.126117 6527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:31.126334 6523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:31.126312 6524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:03:31.127023 6517 server_base.cc:1034] running on GCE node
I20260812 07:03:31.127591 6517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:03:31.128511 6517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:03:31.129740 6517 hybrid_clock.cc:648] HybridClock initialized: now 1786518211129676 us; error 44 us; skew 500 ppm
I20260812 07:03:31.130106 6517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:03:31 dist-test-slave-69rg krb5kdc[6397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518211, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:03:31.133462 6517 init.cc:377] Logged in from keytab as kudu/127.6.61.1@KRBTEST.COM (short username kudu)
I20260812 07:03:31.135169 6517 webserver.cc:460] Webserver started at http://127.6.61.1:33015/ using document root <none> and password file <none>
I20260812 07:03:31.136018 6517 fs_manager.cc:359] Metadata directory not provided
I20260812 07:03:31.136153 6517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:03:31.136490 6517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:03:31.139568 6517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-0/data/instance:
uuid: "1728b3268d6347758595e637aedab6e1"
format_stamp: "Formatted at 2026-08-12 07:03:31 on dist-test-slave-69rg"
server_key: "504e89f02357c507a10f335a953ebfb2"
server_key_iv: "aef685da2a4381372f5fe37651a0a4d7"
server_key_version: "encryptionkey@0"
I20260812 07:03:31.140349 6517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance:
uuid: "1728b3268d6347758595e637aedab6e1"
format_stamp: "Formatted at 2026-08-12 07:03:31 on dist-test-slave-69rg"
server_key: "504e89f02357c507a10f335a953ebfb2"
server_key_iv: "aef685da2a4381372f5fe37651a0a4d7"
server_key_version: "encryptionkey@0"
I20260812 07:03:31.145682 6517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260812 07:03:31.149636 6535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:31.150985 6517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:03:31.151129 6517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-0/wal
uuid: "1728b3268d6347758595e637aedab6e1"
format_stamp: "Formatted at 2026-08-12 07:03:31 on dist-test-slave-69rg"
server_key: "504e89f02357c507a10f335a953ebfb2"
server_key_iv: "aef685da2a4381372f5fe37651a0a4d7"
server_key_version: "encryptionkey@0"
I20260812 07:03:31.151316 6517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:03:31.164984 6517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:03:31.169529 6517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:03:31.169883 6517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:03:31.170885 6517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:03:31.172255 6517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:03:31.172329 6517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:31.172501 6517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:03:31.172586 6517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:31.206539 6517 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.1:34809
I20260812 07:03:31.206579 6658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.1:34809 every 8 connection(s)
I20260812 07:03:31.207907 6517 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
I20260812 07:03:31.208818 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 6517
I20260812 07:03:31.209084 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance
I20260812 07:03:31.209789 6388 external_mini_cluster.cc:1428] Setting key 7a64a3da097def2d8b251970bf149598
Aug 12 07:03:31 dist-test-slave-69rg krb5kdc[6397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518211, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.61.2@KRBTEST.COM; defaulting to no policy
I20260812 07:03:31.229439 6662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:31.210995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:44101 (local address 127.6.61.62:46259)
0812 07:03:31.211579 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:31.211584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:31.212713 (+ 1129us) server_negotiation.cc:407] Connection header received
0812 07:03:31.214630 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:31.214639 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:31.214731 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:31.214859 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:31.216268 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:31.217479 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:31.219111 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:31.219530 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:31.219672 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:31.223248 (+ 3576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:31.223264 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:31.223267 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:31.223299 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:31.226183 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:31.226945 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:31.226949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:31.226952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:31.227070 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:31.227519 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:31.227524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:31.227529 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:31.227819 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:31.227975 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:31.228826 (+ 851us) server_negotiation.cc:299] Negotiation successful
0812 07:03:31.229057 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":346,"threads_started":1}
I20260812 07:03:31.230469 6663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:31.211606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:46259 (local address 127.6.61.1:44101)
0812 07:03:31.212525 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:31.212587 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:31.214362 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:31.215020 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:31.215027 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:31.215527 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:03:31.216094 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:31.216111 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:31.217657 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:31.217661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:31.218961 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:31.218974 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:31.219770 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:31.219779 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:31.220024 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:31.220752 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:03:31.220775 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:03:31.223020 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:03:31.226339 (+ 3319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:31.226349 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:31.226367 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:31.226807 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:31.227187 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:31.227191 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:31.227195 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:31.227396 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:31.227951 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:03:31.227959 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:03:31.228362 (+ 403us) client_negotiation.cc:769] Sending connection context
0812 07:03:31.228698 (+ 336us) client_negotiation.cc:240] Negotiation successful
0812 07:03:31.229257 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":388,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.2@KRBTEST.COM" created.
I20260812 07:03:31.232324 6659 heartbeater.cc:347] Connected to a master server at 127.6.61.62:46259
I20260812 07:03:31.232659 6659 heartbeater.cc:464] Registering TS with master...
I20260812 07:03:31.233414 6659 heartbeater.cc:511] Master 127.6.61.62:46259 requested a full tablet report, sending...
I20260812 07:03:31.235989 6449 ts_manager.cc:194] Registered new tserver with Master: 1728b3268d6347758595e637aedab6e1 (127.6.61.1:34809)
I20260812 07:03:31.237592 6449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.1@KRBTEST.COM'} at 127.6.61.1:44101
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
I20260812 07:03:31.256994 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.2:0
--local_ip_for_outbound_sockets=127.6.61.2
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:46259
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:36761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:03:31.591315 6668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:03:31.591615 6668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:03:31.591789 6668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:03:31.599269 6668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:03:31.599431 6668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:03:31.599606 6668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.2
I20260812 07:03:31.609663 6668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:36761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:46259
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.2
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:03:31.610978 6668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:03:31.612339 6668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:03:31.625411 6674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:31.625370 6675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:31.625875 6677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:03:31.626507 6668 server_base.cc:1034] running on GCE node
I20260812 07:03:31.627095 6668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:03:31.628017 6668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:03:31.629272 6668 hybrid_clock.cc:648] HybridClock initialized: now 1786518211629266 us; error 114 us; skew 500 ppm
I20260812 07:03:31.629606 6668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:03:31 dist-test-slave-69rg krb5kdc[6397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518211, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:03:31.632839 6668 init.cc:377] Logged in from keytab as kudu/127.6.61.2@KRBTEST.COM (short username kudu)
I20260812 07:03:31.634497 6668 webserver.cc:460] Webserver started at http://127.6.61.2:38345/ using document root <none> and password file <none>
I20260812 07:03:31.635465 6668 fs_manager.cc:359] Metadata directory not provided
I20260812 07:03:31.635627 6668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:03:31.636011 6668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:03:31.638974 6668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-1/data/instance:
uuid: "4e0b10a02b8f480e8ff3fc8af14214f5"
format_stamp: "Formatted at 2026-08-12 07:03:31 on dist-test-slave-69rg"
server_key: "8a3ed42e2f93ac863ffbe191c196a406"
server_key_iv: "458d87af45859e5d3b8dff9e6285b450"
server_key_version: "encryptionkey@0"
I20260812 07:03:31.639719 6668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance:
uuid: "4e0b10a02b8f480e8ff3fc8af14214f5"
format_stamp: "Formatted at 2026-08-12 07:03:31 on dist-test-slave-69rg"
server_key: "8a3ed42e2f93ac863ffbe191c196a406"
server_key_iv: "458d87af45859e5d3b8dff9e6285b450"
server_key_version: "encryptionkey@0"
I20260812 07:03:31.644918 6668 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 07:03:31.648458 6684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:31.649966 6668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:03:31.650148 6668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-1/wal
uuid: "4e0b10a02b8f480e8ff3fc8af14214f5"
format_stamp: "Formatted at 2026-08-12 07:03:31 on dist-test-slave-69rg"
server_key: "8a3ed42e2f93ac863ffbe191c196a406"
server_key_iv: "458d87af45859e5d3b8dff9e6285b450"
server_key_version: "encryptionkey@0"
I20260812 07:03:31.650352 6668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:03:31.665412 6668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:03:31.669658 6668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:03:31.670001 6668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:03:31.671020 6668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:03:31.672405 6668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:03:31.672514 6668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:31.672638 6668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:03:31.672717 6668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:31.710183 6668 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.2:41727
I20260812 07:03:31.710245 6814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.2:41727 every 8 connection(s)
I20260812 07:03:31.711530 6668 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
I20260812 07:03:31.716300 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 6668
I20260812 07:03:31.716509 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance
I20260812 07:03:31.717100 6388 external_mini_cluster.cc:1428] Setting key a014fe0405b986ac15d1cbbbebbc8e2c
Aug 12 07:03:31 dist-test-slave-69rg krb5kdc[6397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518211, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:03:31.736240 6662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:31.714457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:46731 (local address 127.6.61.62:46259)
0812 07:03:31.714700 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:31.714707 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:31.715937 (+ 1230us) server_negotiation.cc:407] Connection header received
0812 07:03:31.718580 (+ 2643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:31.718586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:31.718668 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:31.718781 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:31.720810 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:31.721747 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:31.724269 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:31.724713 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:31.724810 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:31.728824 (+ 4014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:31.728833 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:31.728841 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:31.728890 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:31.732419 (+ 3529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:31.733456 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:31.733460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:31.733463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:31.733610 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:31.734242 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:31.734249 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:31.734252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:31.734574 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:31.734724 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:31.735729 (+ 1005us) server_negotiation.cc:299] Negotiation successful
0812 07:03:31.735952 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
WARNING: no policy specified for kudu/127.6.61.3@KRBTEST.COM; defaulting to no policy
I20260812 07:03:31.737763 6817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:31.714912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:46259 (local address 127.6.61.2:46731)
0812 07:03:31.715706 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:31.715751 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:31.718311 (+ 2560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:31.718988 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:31.719000 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:31.719771 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:03:31.720599 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:31.720628 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:31.721952 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:31.721961 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:31.724101 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:31.724130 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:31.724940 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:31.724950 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:31.725274 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:31.725995 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:03:31.726018 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:03:31.728559 (+ 2541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:03:31.732584 (+ 4025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:31.732597 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:31.732626 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:31.733314 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:31.733749 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:31.733757 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:31.733763 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:31.734087 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:31.734720 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:03:31.734732 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:03:31.735228 (+ 496us) client_negotiation.cc:769] Sending connection context
0812 07:03:31.735562 (+ 334us) client_negotiation.cc:240] Negotiation successful
0812 07:03:31.736212 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":320,"threads_started":1}
I20260812 07:03:31.739519 6815 heartbeater.cc:347] Connected to a master server at 127.6.61.62:46259
I20260812 07:03:31.739876 6815 heartbeater.cc:464] Registering TS with master...
I20260812 07:03:31.740818 6815 heartbeater.cc:511] Master 127.6.61.62:46259 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.3@KRBTEST.COM" created.
I20260812 07:03:31.742681 6449 ts_manager.cc:194] Registered new tserver with Master: 4e0b10a02b8f480e8ff3fc8af14214f5 (127.6.61.2:41727)
I20260812 07:03:31.743624 6449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.2@KRBTEST.COM'} at 127.6.61.2:46731
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
I20260812 07:03:31.765609 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.3:0
--local_ip_for_outbound_sockets=127.6.61.3
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:46259
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:36761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:03:32.109774 6820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:03:32.110153 6820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:03:32.110328 6820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:03:32.117594 6820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:03:32.117697 6820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:03:32.117944 6820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.3
I20260812 07:03:32.128526 6820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:36761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:46259
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.3
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:03:32.129890 6820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:03:32.131268 6820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:03:32.144611 6828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:32.144718 6827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:32.144798 6830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:03:32.145854 6820 server_base.cc:1034] running on GCE node
I20260812 07:03:32.146436 6820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:03:32.147459 6820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:03:32.148692 6820 hybrid_clock.cc:648] HybridClock initialized: now 1786518212148626 us; error 57 us; skew 500 ppm
I20260812 07:03:32.149015 6820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:03:32 dist-test-slave-69rg krb5kdc[6397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518212, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:03:32.152290 6820 init.cc:377] Logged in from keytab as kudu/127.6.61.3@KRBTEST.COM (short username kudu)
I20260812 07:03:32.153882 6820 webserver.cc:460] Webserver started at http://127.6.61.3:37311/ using document root <none> and password file <none>
I20260812 07:03:32.154560 6820 fs_manager.cc:359] Metadata directory not provided
I20260812 07:03:32.154735 6820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:03:32.155053 6820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:03:32.158145 6820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-2/data/instance:
uuid: "4c8ef728e45549ac97df1acaa64b1c4f"
format_stamp: "Formatted at 2026-08-12 07:03:32 on dist-test-slave-69rg"
server_key: "d0d4b4be9851d2d37bad92ddc0a40b9e"
server_key_iv: "504bf23dbb887d6480939c8eccf049e2"
server_key_version: "encryptionkey@0"
I20260812 07:03:32.158910 6820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance:
uuid: "4c8ef728e45549ac97df1acaa64b1c4f"
format_stamp: "Formatted at 2026-08-12 07:03:32 on dist-test-slave-69rg"
server_key: "d0d4b4be9851d2d37bad92ddc0a40b9e"
server_key_iv: "504bf23dbb887d6480939c8eccf049e2"
server_key_version: "encryptionkey@0"
I20260812 07:03:32.163964 6820 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 07:03:32.167876 6837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:32.169399 6820 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 07:03:32.169567 6820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-2/wal
uuid: "4c8ef728e45549ac97df1acaa64b1c4f"
format_stamp: "Formatted at 2026-08-12 07:03:32 on dist-test-slave-69rg"
server_key: "d0d4b4be9851d2d37bad92ddc0a40b9e"
server_key_iv: "504bf23dbb887d6480939c8eccf049e2"
server_key_version: "encryptionkey@0"
I20260812 07:03:32.169768 6820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:03:32.223903 6820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:03:32.228166 6820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:03:32.228492 6820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:03:32.229552 6820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:03:32.230921 6820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:03:32.231000 6820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:32.231184 6820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:03:32.231261 6820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:32.240718 6659 heartbeater.cc:503] Master 127.6.61.62:46259 was elected leader, sending a full tablet report...
I20260812 07:03:32.267390 6820 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.3:40607
I20260812 07:03:32.267488 6961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.3:40607 every 8 connection(s)
I20260812 07:03:32.268827 6820 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
I20260812 07:03:32.275436 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 6820
I20260812 07:03:32.275660 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance
I20260812 07:03:32.276134 6388 external_mini_cluster.cc:1428] Setting key fafe9e94b27bf8f95187b8f7ea8e21b4
Aug 12 07:03:32 dist-test-slave-69rg krb5kdc[6397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518212, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:03:32.292152 6965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:32.271620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:55791 (local address 127.6.61.62:46259)
0812 07:03:32.272197 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:32.272202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:32.273062 (+ 860us) server_negotiation.cc:407] Connection header received
0812 07:03:32.275439 (+ 2377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:32.275447 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:32.275553 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:32.275738 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:32.277694 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:32.278710 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:32.280927 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:32.281406 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:32.281529 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:32.285561 (+ 4032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:32.285566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:32.285569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:32.285604 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:32.288583 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:32.289364 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:32.289368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:32.289371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:32.289496 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:32.290011 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:32.290016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:32.290019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:32.290373 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:32.290516 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:32.291503 (+ 987us) server_negotiation.cc:299] Negotiation successful
0812 07:03:32.291723 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":349,"threads_started":1}
I20260812 07:03:32.293496 6966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:32.272114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:46259 (local address 127.6.61.3:55791)
0812 07:03:32.272856 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:32.272935 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:32.275098 (+ 2163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:32.276003 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:32.276012 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:32.276647 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:03:32.277466 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:32.277508 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:32.278880 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:32.278885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:32.280759 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:32.280774 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:32.281715 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:32.281721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:32.282058 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:32.282858 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:03:32.282884 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:03:32.285322 (+ 2438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:03:32.288739 (+ 3417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:32.288748 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:32.288763 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:32.289243 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:32.289605 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:32.289609 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:32.289611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:32.289874 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:32.290515 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:03:32.290522 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:03:32.290889 (+ 367us) client_negotiation.cc:769] Sending connection context
0812 07:03:32.291272 (+ 383us) client_negotiation.cc:240] Negotiation successful
0812 07:03:32.291907 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":302,"threads_started":1}
I20260812 07:03:32.295182 6962 heartbeater.cc:347] Connected to a master server at 127.6.61.62:46259
I20260812 07:03:32.295665 6962 heartbeater.cc:464] Registering TS with master...
I20260812 07:03:32.296635 6962 heartbeater.cc:511] Master 127.6.61.62:46259 requested a full tablet report, sending...
I20260812 07:03:32.298560 6449 ts_manager.cc:194] Registered new tserver with Master: 4c8ef728e45549ac97df1acaa64b1c4f (127.6.61.3:40607)
I20260812 07:03:32.299422 6449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.3@KRBTEST.COM'} at 127.6.61.3:55791
I20260812 07:03:32.306632 6388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 07:03:32 dist-test-slave-69rg krb5kdc[6397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518212, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 07:03:32 dist-test-slave-69rg krb5kdc[6397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518212, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.61.1@KRBTEST.COM
I20260812 07:03:32.363451 6972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:32.345999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51930 (local address 127.6.61.1:34809)
0812 07:03:32.346783 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:32.346798 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:32.346838 (+ 40us) server_negotiation.cc:407] Connection header received
0812 07:03:32.346926 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:32.346934 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:32.347179 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:32.347371 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:32.349087 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:32.350273 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:32.351699 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:32.352383 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:32.352539 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:32.357356 (+ 4817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:32.357369 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:32.357386 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:32.357428 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:32.360446 (+ 3018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:32.361178 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:32.361186 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:32.361188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:32.361312 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:32.361753 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:32.361758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:32.361761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:32.362192 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:32.362509 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:32.362801 (+ 292us) server_negotiation.cc:299] Negotiation successful
0812 07:03:32.363081 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":335,"threads_started":1}
W20260812 07:03:32.364880 6568 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51930
Aug 12 07:03:32 dist-test-slave-69rg krb5kdc[6397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 07:03:32 dist-test-slave-69rg krb5kdc[6397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.1@KRBTEST.COM
I20260812 07:03:32.420153 6972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:32.406286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51944 (local address 127.6.61.1:34809)
0812 07:03:32.406626 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:32.406631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:32.406768 (+ 137us) server_negotiation.cc:407] Connection header received
0812 07:03:32.406920 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:32.406924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:32.406994 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:32.407081 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:32.408533 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:32.409270 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:32.410574 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:32.411018 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:32.411122 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:32.415181 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:32.415186 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:32.415190 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:32.415222 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:32.417765 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:32.418462 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:32.418466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:32.418468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:32.418589 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:32.418921 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:32.418924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:32.418926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:32.419162 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:32.419341 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:32.419609 (+ 268us) server_negotiation.cc:299] Negotiation successful
0812 07:03:32.419796 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260812 07:03:32.421926 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 6517
I20260812 07:03:32.434412 6972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:32.422909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51954 (local address 127.6.61.1:34809)
0812 07:03:32.423067 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:32.423071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:32.423503 (+ 432us) server_negotiation.cc:407] Connection header received
0812 07:03:32.423618 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:32.423621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:32.423675 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:32.423767 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:32.424888 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:32.425591 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:32.426820 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:32.427225 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:32.427319 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:32.429554 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:32.429558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:32.429562 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:32.429589 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:32.432130 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:32.432715 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:32.432718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:32.432720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:32.432859 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:32.433213 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:32.433216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:32.433218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:32.433481 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:32.433655 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:32.433997 (+ 342us) server_negotiation.cc:299] Negotiation successful
0812 07:03:32.434146 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 07:03:32.435177 6654 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:03:32.519361 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 6517
I20260812 07:03:32.540560 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 6668
Aug 12 07:03:32 dist-test-slave-69rg krb5kdc[6397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.2@KRBTEST.COM
I20260812 07:03:32.558267 6979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:32.541517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58760 (local address 127.6.61.2:41727)
0812 07:03:32.542262 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:32.542270 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:32.542296 (+ 26us) server_negotiation.cc:407] Connection header received
0812 07:03:32.542364 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:32.542371 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:32.542595 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:32.542783 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:32.544052 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:32.545162 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:32.546610 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:32.547195 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:32.547348 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:32.551752 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:32.551765 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:32.551783 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:32.551822 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:32.555395 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:32.556109 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:32.556116 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:32.556119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:32.556266 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:32.556755 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:32.556759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:32.556761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:32.557130 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:32.557374 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:32.557666 (+ 292us) server_negotiation.cc:299] Negotiation successful
0812 07:03:32.557943 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":472,"threads_started":1}
I20260812 07:03:32.559594 6809 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:03:32.645247 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 6668
I20260812 07:03:32.666006 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 6820
Aug 12 07:03:32 dist-test-slave-69rg krb5kdc[6397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.3@KRBTEST.COM
I20260812 07:03:32.683748 6981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:32.667156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55920 (local address 127.6.61.3:40607)
0812 07:03:32.667762 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:32.667770 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:32.667801 (+ 31us) server_negotiation.cc:407] Connection header received
0812 07:03:32.667865 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:32.667873 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:32.668092 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:32.668273 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:32.669772 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:32.670868 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:32.672271 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:32.672855 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:32.673001 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:32.677486 (+ 4485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:32.677497 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:32.677513 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:32.677560 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:32.680612 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:32.681383 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:32.681391 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:32.681393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:32.681527 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:32.682005 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:32.682011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:32.682016 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:32.682387 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:32.682732 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:32.683076 (+ 344us) server_negotiation.cc:299] Negotiation successful
0812 07:03:32.683367 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":263,"threads_started":1}
I20260812 07:03:32.685122 6955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:03:32.768424 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 6820
I20260812 07:03:32.788877 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 6411
I20260812 07:03:32.790239 6485 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:03:32.893209 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 6411
2026-08-12T07:03:32Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4174 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:03:32 dist-test-slave-69rg krb5kdc[6986](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:03:32 dist-test-slave-69rg krb5kdc[6986](info): set up 2 sockets
Aug 12 07:03:32 dist-test-slave-69rg krb5kdc[6986](info): commencing operation
krb5kdc: starting...
W20260812 07:03:34.062014 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:03:34 dist-test-slave-69rg krb5kdc[6986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:03:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:03:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
I20260812 07:03:34.266426 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:37143
--webserver_interface=127.6.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:34881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:37143
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:03:34.596366 7002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:03:34.596692 7002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:03:34.596846 7002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:03:34.604206 7002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:03:34.604350 7002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:03:34.604452 7002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:03:34.604555 7002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:03:34.604638 7002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:03:34.614670 7002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:34881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:37143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:37143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:03:34.616034 7002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:03:34.617336 7002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:03:34.629639 7007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:34.629738 7010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:34.629724 7008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:03:34.630504 7002 server_base.cc:1034] running on GCE node
I20260812 07:03:34.631326 7002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:03:34.632722 7002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:03:34.633914 7002 hybrid_clock.cc:648] HybridClock initialized: now 1786518214633858 us; error 53 us; skew 500 ppm
I20260812 07:03:34.634276 7002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:03:34 dist-test-slave-69rg krb5kdc[6986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518214, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:03:34.642602 7002 init.cc:377] Logged in from keytab as kudu/127.6.61.62@KRBTEST.COM (short username kudu)
I20260812 07:03:34.644207 7002 webserver.cc:460] Webserver started at http://127.6.61.62:44899/ using document root <none> and password file <none>
I20260812 07:03:34.644994 7002 fs_manager.cc:359] Metadata directory not provided
I20260812 07:03:34.645100 7002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:03:34.645381 7002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:03:34.648450 7002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "765de4b5fed94b03a09d3b7ab83a8263"
format_stamp: "Formatted at 2026-08-12 07:03:34 on dist-test-slave-69rg"
server_key: "6359ef6cac44cd1121db19caf919e684"
server_key_iv: "353e4add5f75a43e63459219d886b036"
server_key_version: "encryptionkey@0"
I20260812 07:03:34.649186 7002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "765de4b5fed94b03a09d3b7ab83a8263"
format_stamp: "Formatted at 2026-08-12 07:03:34 on dist-test-slave-69rg"
server_key: "6359ef6cac44cd1121db19caf919e684"
server_key_iv: "353e4add5f75a43e63459219d886b036"
server_key_version: "encryptionkey@0"
I20260812 07:03:34.655109 7002 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260812 07:03:34.659413 7019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:34.660842 7002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260812 07:03:34.661038 7002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "765de4b5fed94b03a09d3b7ab83a8263"
format_stamp: "Formatted at 2026-08-12 07:03:34 on dist-test-slave-69rg"
server_key: "6359ef6cac44cd1121db19caf919e684"
server_key_iv: "353e4add5f75a43e63459219d886b036"
server_key_version: "encryptionkey@0"
I20260812 07:03:34.661283 7002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:03:34.678480 7002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:03:34.682799 7002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:03:34.683111 7002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:03:34.703138 7002 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:37143
I20260812 07:03:34.703234 7081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:37143 every 8 connection(s)
I20260812 07:03:34.704644 7002 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:03:34.706344 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 7002
I20260812 07:03:34.706586 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:03:34.707475 6388 external_mini_cluster.cc:1428] Setting key 4973c546866ee73b0bf133e0d333ccae
I20260812 07:03:34.709640 7082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 07:03:34 dist-test-slave-69rg krb5kdc[6986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:03:34.723377 7082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 765de4b5fed94b03a09d3b7ab83a8263: Bootstrap starting.
I20260812 07:03:34.727599 7082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 765de4b5fed94b03a09d3b7ab83a8263: Neither blocks nor log segments found. Creating new log.
I20260812 07:03:34.728945 7082 log.cc:824] T 00000000000000000000000000000000 P 765de4b5fed94b03a09d3b7ab83a8263: Log is configured to *not* fsync() on all Append() calls
I20260812 07:03:34.732694 7082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 765de4b5fed94b03a09d3b7ab83a8263: No bootstrap required, opened a new log
I20260812 07:03:34.733392 7085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:34.709774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41344 (local address 127.6.61.62:37143)
0812 07:03:34.710630 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:34.710646 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:34.710699 (+ 53us) server_negotiation.cc:407] Connection header received
0812 07:03:34.712831 (+ 2132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:34.712860 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:34.713395 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:34.713906 (+ 511us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:34.715340 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:34.717406 (+ 2066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:34.718792 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:34.719842 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:34.720041 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:34.724247 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:34.724258 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:34.724278 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:34.724322 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:34.728500 (+ 4178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:34.729314 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:34.729322 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:34.729331 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:34.729560 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:34.730091 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:34.730096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:34.730099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:34.730783 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:34.731092 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:34.731575 (+ 483us) server_negotiation.cc:299] Negotiation successful
0812 07:03:34.732032 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":314,"threads_started":1}
I20260812 07:03:34.741304 7082 raft_consensus.cc:348] T 00000000000000000000000000000000 P 765de4b5fed94b03a09d3b7ab83a8263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "765de4b5fed94b03a09d3b7ab83a8263" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 37143 } }
I20260812 07:03:34.741667 7082 raft_consensus.cc:374] T 00000000000000000000000000000000 P 765de4b5fed94b03a09d3b7ab83a8263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:03:34.741748 7082 raft_consensus.cc:729] T 00000000000000000000000000000000 P 765de4b5fed94b03a09d3b7ab83a8263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 765de4b5fed94b03a09d3b7ab83a8263, State: Initialized, Role: FOLLOWER
I20260812 07:03:34.742360 7082 consensus_queue.cc:260] T 00000000000000000000000000000000 P 765de4b5fed94b03a09d3b7ab83a8263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "765de4b5fed94b03a09d3b7ab83a8263" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 37143 } }
I20260812 07:03:34.742568 7082 raft_consensus.cc:388] T 00000000000000000000000000000000 P 765de4b5fed94b03a09d3b7ab83a8263 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:03:34.742666 7082 raft_consensus.cc:482] T 00000000000000000000000000000000 P 765de4b5fed94b03a09d3b7ab83a8263 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:03:34.742799 7082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 765de4b5fed94b03a09d3b7ab83a8263 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:03:34.745139 7082 raft_consensus.cc:504] T 00000000000000000000000000000000 P 765de4b5fed94b03a09d3b7ab83a8263 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "765de4b5fed94b03a09d3b7ab83a8263" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 37143 } }
I20260812 07:03:34.745579 7082 leader_election.cc:302] T 00000000000000000000000000000000 P 765de4b5fed94b03a09d3b7ab83a8263 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 765de4b5fed94b03a09d3b7ab83a8263; no voters:
I20260812 07:03:34.746186 7082 leader_election.cc:288] T 00000000000000000000000000000000 P 765de4b5fed94b03a09d3b7ab83a8263 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:03:34.746354 7087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 765de4b5fed94b03a09d3b7ab83a8263 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:03:34.747030 7087 raft_consensus.cc:686] T 00000000000000000000000000000000 P 765de4b5fed94b03a09d3b7ab83a8263 [term 1 LEADER]: Becoming Leader. State: Replica: 765de4b5fed94b03a09d3b7ab83a8263, State: Running, Role: LEADER
I20260812 07:03:34.747539 7087 consensus_queue.cc:237] T 00000000000000000000000000000000 P 765de4b5fed94b03a09d3b7ab83a8263 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "765de4b5fed94b03a09d3b7ab83a8263" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 37143 } }
I20260812 07:03:34.748287 7082 sys_catalog.cc:564] T 00000000000000000000000000000000 P 765de4b5fed94b03a09d3b7ab83a8263 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:03:34.751056 7089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 765de4b5fed94b03a09d3b7ab83a8263 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 765de4b5fed94b03a09d3b7ab83a8263. Latest consensus state: current_term: 1 leader_uuid: "765de4b5fed94b03a09d3b7ab83a8263" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "765de4b5fed94b03a09d3b7ab83a8263" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 37143 } } }
I20260812 07:03:34.750954 7088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 765de4b5fed94b03a09d3b7ab83a8263 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "765de4b5fed94b03a09d3b7ab83a8263" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "765de4b5fed94b03a09d3b7ab83a8263" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 37143 } } }
I20260812 07:03:34.751351 7089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 765de4b5fed94b03a09d3b7ab83a8263 [sys.catalog]: This master's current role is: LEADER
I20260812 07:03:34.751351 7088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 765de4b5fed94b03a09d3b7ab83a8263 [sys.catalog]: This master's current role is: LEADER
I20260812 07:03:34.753744 7096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:03:34.758371 7096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:03:34.767778 7096 catalog_manager.cc:1347] Generated new cluster ID: c0bfe45de1e74cb9be63f32378f399a4
I20260812 07:03:34.767925 7096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:03:34.780417 7096 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:03:34.781805 7096 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:03:34.788324 7096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 765de4b5fed94b03a09d3b7ab83a8263: Generated new TSK 0
I20260812 07:03:34.789324 7096 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
I20260812 07:03:34.847697 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.1:0
--local_ip_for_outbound_sockets=127.6.61.1
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:37143
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:34881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:03:35.180626 7111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:03:35.180915 7111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:03:35.181082 7111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:03:35.188553 7111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:03:35.188676 7111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:03:35.188861 7111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.1
I20260812 07:03:35.199062 7111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:34881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:37143
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.1
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:03:35.200506 7111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:03:35.201949 7111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:03:35.216012 7117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:35.215970 7116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:35.216055 7119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:03:35.217428 7111 server_base.cc:1034] running on GCE node
I20260812 07:03:35.218132 7111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:03:35.219133 7111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:03:35.220350 7111 hybrid_clock.cc:648] HybridClock initialized: now 1786518215220292 us; error 51 us; skew 500 ppm
I20260812 07:03:35.220683 7111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:03:35 dist-test-slave-69rg krb5kdc[6986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518215, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:03:35.223984 7111 init.cc:377] Logged in from keytab as kudu/127.6.61.1@KRBTEST.COM (short username kudu)
I20260812 07:03:35.225499 7111 webserver.cc:460] Webserver started at http://127.6.61.1:34263/ using document root <none> and password file <none>
I20260812 07:03:35.226253 7111 fs_manager.cc:359] Metadata directory not provided
I20260812 07:03:35.226366 7111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:03:35.226708 7111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:03:35.229712 7111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-0/data/instance:
uuid: "39ca7c5a5e4646e694c94effb0a7c314"
format_stamp: "Formatted at 2026-08-12 07:03:35 on dist-test-slave-69rg"
server_key: "d824c6d6f63d34eb20336ef94b97d6ae"
server_key_iv: "5d23a17ada259aa0740cd65c3981a5ea"
server_key_version: "encryptionkey@0"
I20260812 07:03:35.230607 7111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance:
uuid: "39ca7c5a5e4646e694c94effb0a7c314"
format_stamp: "Formatted at 2026-08-12 07:03:35 on dist-test-slave-69rg"
server_key: "d824c6d6f63d34eb20336ef94b97d6ae"
server_key_iv: "5d23a17ada259aa0740cd65c3981a5ea"
server_key_version: "encryptionkey@0"
I20260812 07:03:35.236377 7111 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 07:03:35.240069 7128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:35.241449 7111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 07:03:35.241640 7111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-0/wal
uuid: "39ca7c5a5e4646e694c94effb0a7c314"
format_stamp: "Formatted at 2026-08-12 07:03:35 on dist-test-slave-69rg"
server_key: "d824c6d6f63d34eb20336ef94b97d6ae"
server_key_iv: "5d23a17ada259aa0740cd65c3981a5ea"
server_key_version: "encryptionkey@0"
I20260812 07:03:35.241873 7111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:03:35.264600 7111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:03:35.269601 7111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:03:35.269980 7111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:03:35.271030 7111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:03:35.272475 7111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:03:35.272594 7111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:35.272728 7111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:03:35.272792 7111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:35.307722 7111 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.1:37607
I20260812 07:03:35.307816 7259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.1:37607 every 8 connection(s)
I20260812 07:03:35.309090 7111 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
I20260812 07:03:35.318780 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 7111
I20260812 07:03:35.319064 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance
I20260812 07:03:35.319844 6388 external_mini_cluster.cc:1428] Setting key f20eecfcdc171ec10a1944d361bdfc84
Aug 12 07:03:35 dist-test-slave-69rg krb5kdc[6986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518215, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:03:35.333580 7262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:35.311831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:39563 (local address 127.6.61.62:37143)
0812 07:03:35.312421 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:35.312426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:35.313439 (+ 1013us) server_negotiation.cc:407] Connection header received
0812 07:03:35.316033 (+ 2594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:35.316038 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:35.316134 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:35.316251 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:35.318860 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:35.319784 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:35.321877 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:35.322296 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:35.322452 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:35.326779 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:35.326787 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:35.326790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:35.326829 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:35.329977 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:35.330866 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:35.330880 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:35.330883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:35.331019 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:35.331549 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:35.331553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:35.331555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:35.331866 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:35.332046 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:35.332903 (+ 857us) server_negotiation.cc:299] Negotiation successful
0812 07:03:35.333164 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":313,"threads_started":1}
I20260812 07:03:35.334708 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:35.312408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:37143 (local address 127.6.61.1:39563)
0812 07:03:35.313232 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:35.313280 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:35.315586 (+ 2306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:35.316657 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:35.316667 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:35.317396 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:03:35.318565 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:35.318600 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:35.319990 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:35.319996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:35.321541 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:35.321555 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:35.322556 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:35.322566 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:35.322839 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:35.323628 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:03:35.323655 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:03:35.326490 (+ 2835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:03:35.330153 (+ 3663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:35.330165 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:35.330187 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:35.330708 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:35.331155 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:35.331161 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:35.331165 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:35.331397 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:35.332016 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:03:35.332025 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:03:35.332442 (+ 417us) client_negotiation.cc:769] Sending connection context
0812 07:03:35.332757 (+ 315us) client_negotiation.cc:240] Negotiation successful
0812 07:03:35.333390 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":308,"threads_started":1}
I20260812 07:03:35.336521 7260 heartbeater.cc:347] Connected to a master server at 127.6.61.62:37143
I20260812 07:03:35.336882 7260 heartbeater.cc:464] Registering TS with master...
I20260812 07:03:35.337674 7260 heartbeater.cc:511] Master 127.6.61.62:37143 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.61.2@KRBTEST.COM; defaulting to no policy
I20260812 07:03:35.340098 7038 ts_manager.cc:194] Registered new tserver with Master: 39ca7c5a5e4646e694c94effb0a7c314 (127.6.61.1:37607)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.2@KRBTEST.COM" created.
I20260812 07:03:35.341944 7038 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.1@KRBTEST.COM'} at 127.6.61.1:39563
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
I20260812 07:03:35.367470 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.2:0
--local_ip_for_outbound_sockets=127.6.61.2
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:37143
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:34881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:03:35.703997 7268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:03:35.704290 7268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:03:35.704459 7268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:03:35.711916 7268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:03:35.712040 7268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:03:35.712240 7268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.2
I20260812 07:03:35.722772 7268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:34881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:37143
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.2
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:03:35.724310 7268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:03:35.725718 7268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:03:35.739694 7275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:35.740163 7278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:35.740202 7274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:03:35.741034 7268 server_base.cc:1034] running on GCE node
I20260812 07:03:35.741712 7268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:03:35.742785 7268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:03:35.743989 7268 hybrid_clock.cc:648] HybridClock initialized: now 1786518215743917 us; error 66 us; skew 500 ppm
I20260812 07:03:35.744340 7268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:03:35 dist-test-slave-69rg krb5kdc[6986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518215, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:03:35.747952 7268 init.cc:377] Logged in from keytab as kudu/127.6.61.2@KRBTEST.COM (short username kudu)
I20260812 07:03:35.749632 7268 webserver.cc:460] Webserver started at http://127.6.61.2:45361/ using document root <none> and password file <none>
I20260812 07:03:35.750489 7268 fs_manager.cc:359] Metadata directory not provided
I20260812 07:03:35.750623 7268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:03:35.750977 7268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:03:35.754168 7268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-1/data/instance:
uuid: "6bc7d6db2c3b42b7b1dc3f73beab063f"
format_stamp: "Formatted at 2026-08-12 07:03:35 on dist-test-slave-69rg"
server_key: "26919e8148d1330247518bade7f7ce63"
server_key_iv: "d5af5cc7486143361812747ba5e9b208"
server_key_version: "encryptionkey@0"
I20260812 07:03:35.754994 7268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance:
uuid: "6bc7d6db2c3b42b7b1dc3f73beab063f"
format_stamp: "Formatted at 2026-08-12 07:03:35 on dist-test-slave-69rg"
server_key: "26919e8148d1330247518bade7f7ce63"
server_key_iv: "d5af5cc7486143361812747ba5e9b208"
server_key_version: "encryptionkey@0"
I20260812 07:03:35.760342 7268 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 07:03:35.763907 7285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:35.765499 7268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 07:03:35.765676 7268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-1/wal
uuid: "6bc7d6db2c3b42b7b1dc3f73beab063f"
format_stamp: "Formatted at 2026-08-12 07:03:35 on dist-test-slave-69rg"
server_key: "26919e8148d1330247518bade7f7ce63"
server_key_iv: "d5af5cc7486143361812747ba5e9b208"
server_key_version: "encryptionkey@0"
I20260812 07:03:35.765950 7268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:03:35.783699 7268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:03:35.787798 7268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:03:35.788121 7268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:03:35.789325 7268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:03:35.790827 7268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:03:35.790903 7268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:35.791071 7268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:03:35.791124 7268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:35.826275 7268 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.2:37361
I20260812 07:03:35.826404 7410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.2:37361 every 8 connection(s)
I20260812 07:03:35.827656 7268 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
I20260812 07:03:35.828595 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 7268
I20260812 07:03:35.828814 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance
I20260812 07:03:35.829458 6388 external_mini_cluster.cc:1428] Setting key 0cbbb4ab62fb19286d7ba187cddde449
Aug 12 07:03:35 dist-test-slave-69rg krb5kdc[6986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518215, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.61.3@KRBTEST.COM; defaulting to no policy
I20260812 07:03:35.852528 7262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:35.830811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:56759 (local address 127.6.61.62:37143)
0812 07:03:35.831016 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:35.831023 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:35.832367 (+ 1344us) server_negotiation.cc:407] Connection header received
0812 07:03:35.834301 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:35.834310 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:35.834413 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:35.834532 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:35.836392 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:35.837532 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:35.840157 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:35.840571 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:35.840669 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:35.845017 (+ 4348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:35.845026 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:35.845034 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:35.845084 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:35.849026 (+ 3942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:35.849882 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:35.849889 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:35.849892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:35.850065 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:35.850507 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:35.850515 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:35.850521 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:35.850897 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:35.851071 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:35.851828 (+ 757us) server_negotiation.cc:299] Negotiation successful
0812 07:03:35.852169 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 07:03:35.853443 7413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:35.831330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:37143 (local address 127.6.61.2:56759)
0812 07:03:35.832184 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:35.832232 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:35.834022 (+ 1790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:35.834827 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:35.834836 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:35.835602 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:03:35.836197 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:35.836216 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:35.837724 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:35.837732 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:35.840010 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:35.840029 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:35.840814 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:35.840824 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:35.841156 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:35.841954 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:03:35.841997 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:03:35.844702 (+ 2705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:03:35.849174 (+ 4472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:35.849182 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:35.849197 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:35.849714 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:35.850181 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:35.850184 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:35.850186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:35.850368 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:35.851049 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:03:35.851059 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:03:35.851414 (+ 355us) client_negotiation.cc:769] Sending connection context
0812 07:03:35.851638 (+ 224us) client_negotiation.cc:240] Negotiation successful
0812 07:03:35.852141 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":383,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.3@KRBTEST.COM" created.
I20260812 07:03:35.855063 7411 heartbeater.cc:347] Connected to a master server at 127.6.61.62:37143
I20260812 07:03:35.855427 7411 heartbeater.cc:464] Registering TS with master...
I20260812 07:03:35.856370 7411 heartbeater.cc:511] Master 127.6.61.62:37143 requested a full tablet report, sending...
I20260812 07:03:35.858103 7038 ts_manager.cc:194] Registered new tserver with Master: 6bc7d6db2c3b42b7b1dc3f73beab063f (127.6.61.2:37361)
I20260812 07:03:35.858960 7038 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.2@KRBTEST.COM'} at 127.6.61.2:56759
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
I20260812 07:03:35.877755 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.3:0
--local_ip_for_outbound_sockets=127.6.61.3
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:37143
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:34881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:03:36.242532 7416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:03:36.242847 7416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:03:36.243022 7416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:03:36.250510 7416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:03:36.250654 7416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:03:36.250834 7416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.3
I20260812 07:03:36.260860 7416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:34881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:37143
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.3
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:03:36.262395 7416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:03:36.263854 7416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:03:36.277133 7427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:36.277288 7423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:36.277314 7424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:03:36.278373 7416 server_base.cc:1034] running on GCE node
I20260812 07:03:36.278964 7416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:03:36.279940 7416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:03:36.281168 7416 hybrid_clock.cc:648] HybridClock initialized: now 1786518216281128 us; error 44 us; skew 500 ppm
I20260812 07:03:36.281500 7416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:03:36 dist-test-slave-69rg krb5kdc[6986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518216, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:03:36.284679 7416 init.cc:377] Logged in from keytab as kudu/127.6.61.3@KRBTEST.COM (short username kudu)
I20260812 07:03:36.286164 7416 webserver.cc:460] Webserver started at http://127.6.61.3:37491/ using document root <none> and password file <none>
I20260812 07:03:36.286866 7416 fs_manager.cc:359] Metadata directory not provided
I20260812 07:03:36.287012 7416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:03:36.287330 7416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:03:36.290262 7416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-2/data/instance:
uuid: "0f00c0808cfa45e799c0974c7cd2196f"
format_stamp: "Formatted at 2026-08-12 07:03:36 on dist-test-slave-69rg"
server_key: "0f8d8af694c113955fed736709a9a534"
server_key_iv: "870c445ac88b646d66902565eceaa2cf"
server_key_version: "encryptionkey@0"
I20260812 07:03:36.291062 7416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance:
uuid: "0f00c0808cfa45e799c0974c7cd2196f"
format_stamp: "Formatted at 2026-08-12 07:03:36 on dist-test-slave-69rg"
server_key: "0f8d8af694c113955fed736709a9a534"
server_key_iv: "870c445ac88b646d66902565eceaa2cf"
server_key_version: "encryptionkey@0"
I20260812 07:03:36.296339 7416 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 07:03:36.299808 7436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:36.301160 7416 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 07:03:36.301378 7416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-2/wal
uuid: "0f00c0808cfa45e799c0974c7cd2196f"
format_stamp: "Formatted at 2026-08-12 07:03:36 on dist-test-slave-69rg"
server_key: "0f8d8af694c113955fed736709a9a534"
server_key_iv: "870c445ac88b646d66902565eceaa2cf"
server_key_version: "encryptionkey@0"
I20260812 07:03:36.301605 7416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:03:36.320937 7416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:03:36.325126 7416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:03:36.325443 7416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:03:36.326488 7416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:03:36.327925 7416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:03:36.328032 7416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:36.328198 7416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:03:36.328274 7416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:36.345314 7260 heartbeater.cc:503] Master 127.6.61.62:37143 was elected leader, sending a full tablet report...
I20260812 07:03:36.363790 7416 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.3:34387
I20260812 07:03:36.363837 7568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.3:34387 every 8 connection(s)
I20260812 07:03:36.365272 7416 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
I20260812 07:03:36.368605 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 7416
I20260812 07:03:36.368809 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance
I20260812 07:03:36.369412 6388 external_mini_cluster.cc:1428] Setting key 25a7a0dcbeeb39bf75c7594d23838f1e
Aug 12 07:03:36 dist-test-slave-69rg krb5kdc[6986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518216, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:03:36.387461 7572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:36.368339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:46623 (local address 127.6.61.62:37143)
0812 07:03:36.368834 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:36.368839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:36.370231 (+ 1392us) server_negotiation.cc:407] Connection header received
0812 07:03:36.372366 (+ 2135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:36.372373 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:36.372474 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:36.372602 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:36.374305 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.375090 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.376686 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.377083 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.377226 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:36.380953 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:36.380958 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:36.380962 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:36.380996 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:36.384169 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:36.385141 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:36.385147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:36.385150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:36.385321 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:36.385908 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:36.385912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:36.385914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:36.386194 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:36.386317 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:36.387008 (+ 691us) server_negotiation.cc:299] Negotiation successful
0812 07:03:36.387187 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":279,"threads_started":1}
I20260812 07:03:36.388428 7573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:36.369078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:37143 (local address 127.6.61.3:46623)
0812 07:03:36.370006 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:36.370078 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:36.372095 (+ 2017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:36.372885 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:36.372900 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:36.373462 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:03:36.374128 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:36.374149 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.375260 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.375266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:36.376510 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:36.376525 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.377332 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.377340 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:36.377576 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:36.378359 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:03:36.378387 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:03:36.380691 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:03:36.384342 (+ 3651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:36.384351 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:36.384370 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:36.385000 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:36.385465 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:36.385469 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:36.385471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:36.385754 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:36.386318 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:03:36.386324 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:03:36.386650 (+ 326us) client_negotiation.cc:769] Sending connection context
0812 07:03:36.386893 (+ 243us) client_negotiation.cc:240] Negotiation successful
0812 07:03:36.387300 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":382,"threads_started":1}
I20260812 07:03:36.390029 7569 heartbeater.cc:347] Connected to a master server at 127.6.61.62:37143
I20260812 07:03:36.390348 7569 heartbeater.cc:464] Registering TS with master...
I20260812 07:03:36.391093 7569 heartbeater.cc:511] Master 127.6.61.62:37143 requested a full tablet report, sending...
I20260812 07:03:36.392627 7038 ts_manager.cc:194] Registered new tserver with Master: 0f00c0808cfa45e799c0974c7cd2196f (127.6.61.3:34387)
I20260812 07:03:36.393395 7038 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.3@KRBTEST.COM'} at 127.6.61.3:46623
I20260812 07:03:36.397981 6388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 07:03:36.417387 7572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:36.404365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41352 (local address 127.6.61.62:37143)
0812 07:03:36.404555 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:36.404559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:36.405122 (+ 563us) server_negotiation.cc:407] Connection header received
0812 07:03:36.405347 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:36.405351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:36.405416 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:36.405519 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:36.407163 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.407883 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.409180 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.409581 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.409720 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:36.412134 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:36.412139 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:36.412143 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:36.412176 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:36.414749 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:36.415416 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:36.415421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:36.415423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:36.415560 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:36.415943 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:36.415946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:36.415948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:36.416302 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:36.416495 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:36.416823 (+ 328us) server_negotiation.cc:299] Negotiation successful
0812 07:03:36.416994 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 07:03:36.424065 7038 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:03:36.431030 7038 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:03:36.465199 7583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:36.443131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:34387 (local address 127.0.0.1:58652)
0812 07:03:36.455857 (+ 12726us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:36.455903 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:36.456118 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:36.457553 (+ 1435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:36.457562 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:36.457605 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:03:36.458330 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:36.458346 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.460286 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.460293 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:36.462373 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:36.462391 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.464194 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.464200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:36.464396 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:36.464459 (+ 63us) client_negotiation.cc:769] Sending connection context
0812 07:03:36.464590 (+ 131us) client_negotiation.cc:240] Negotiation successful
0812 07:03:36.464785 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12417,"thread_start_us":12249,"threads_started":1}
I20260812 07:03:36.466341 7587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:36.444576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58370 (local address 127.6.61.2:37361)
0812 07:03:36.458103 (+ 13527us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:36.458113 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:36.458141 (+ 28us) server_negotiation.cc:407] Connection header received
0812 07:03:36.458251 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:36.458258 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:36.458466 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:36.458691 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:03:36.459488 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.461065 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.463347 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.464625 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.464763 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:36.464943 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:36.465787 (+ 844us) server_negotiation.cc:299] Negotiation successful
0812 07:03:36.465996 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13363,"thread_start_us":13247,"threads_started":1}
I20260812 07:03:36.466691 7584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:36.443474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58652 (local address 127.6.61.3:34387)
0812 07:03:36.456706 (+ 13232us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:36.456719 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:36.456759 (+ 40us) server_negotiation.cc:407] Connection header received
0812 07:03:36.456876 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:36.456887 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:36.457171 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:36.457382 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:03:36.458550 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.460127 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.462564 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.464001 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.464142 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:36.464393 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:36.466072 (+ 1679us) server_negotiation.cc:299] Negotiation successful
0812 07:03:36.466249 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12991,"thread_start_us":12821,"threads_started":1}
I20260812 07:03:36.466638 7589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:36.443959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:37361 (local address 127.0.0.1:58370)
0812 07:03:36.457546 (+ 13587us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:36.457563 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:36.457673 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:36.458671 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:36.458676 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:36.458702 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:03:36.459104 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:36.459127 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.461219 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.461225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:36.463187 (+ 1962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:36.463200 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.464846 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.464851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:36.465040 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:36.465070 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 07:03:36.465167 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 07:03:36.465309 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13474,"mutex_wait_us":31,"thread_start_us":14073,"threads_started":1}
I20260812 07:03:36.467458 7590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:36.445535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54054 (local address 127.6.61.1:37607)
0812 07:03:36.457074 (+ 11539us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:36.457087 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:36.458484 (+ 1397us) server_negotiation.cc:407] Connection header received
0812 07:03:36.458783 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:36.458793 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:36.459011 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:36.459208 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:03:36.459965 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.461434 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.463890 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.465406 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.465538 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:36.465766 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:36.466884 (+ 1118us) server_negotiation.cc:299] Negotiation successful
0812 07:03:36.467063 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11345,"thread_start_us":11189,"threads_started":1}
I20260812 07:03:36.466765 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:36.444696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:37607 (local address 127.0.0.1:54054)
0812 07:03:36.458387 (+ 13691us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:36.458404 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:36.458575 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:36.459347 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:36.459351 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:36.459374 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:03:36.459781 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:36.459794 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.461577 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.461583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:36.463700 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:36.463716 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.465755 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.465762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:36.465947 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:36.465980 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 07:03:36.466072 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 07:03:36.466204 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13571,"mutex_wait_us":55,"thread_start_us":12565,"threads_started":1}
I20260812 07:03:36.471022 7187 tablet_service.cc:1467] Processing CreateTablet for tablet da92cdb1e42f4d4f853281aadf067187 (DEFAULT_TABLE table=test-table [id=9f7dafe8aa374c15869a0547ee7f0840]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:03:36.471042 7496 tablet_service.cc:1467] Processing CreateTablet for tablet da92cdb1e42f4d4f853281aadf067187 (DEFAULT_TABLE table=test-table [id=9f7dafe8aa374c15869a0547ee7f0840]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:03:36.471040 7339 tablet_service.cc:1467] Processing CreateTablet for tablet da92cdb1e42f4d4f853281aadf067187 (DEFAULT_TABLE table=test-table [id=9f7dafe8aa374c15869a0547ee7f0840]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:03:36.472532 7187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da92cdb1e42f4d4f853281aadf067187. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:03:36.472563 7496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da92cdb1e42f4d4f853281aadf067187. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:03:36.473014 7339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da92cdb1e42f4d4f853281aadf067187. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:03:36.483893 7591 tablet_bootstrap.cc:492] T da92cdb1e42f4d4f853281aadf067187 P 39ca7c5a5e4646e694c94effb0a7c314: Bootstrap starting.
I20260812 07:03:36.487553 7591 tablet_bootstrap.cc:654] T da92cdb1e42f4d4f853281aadf067187 P 39ca7c5a5e4646e694c94effb0a7c314: Neither blocks nor log segments found. Creating new log.
I20260812 07:03:36.488595 7591 log.cc:824] T da92cdb1e42f4d4f853281aadf067187 P 39ca7c5a5e4646e694c94effb0a7c314: Log is configured to *not* fsync() on all Append() calls
I20260812 07:03:36.490084 7593 tablet_bootstrap.cc:492] T da92cdb1e42f4d4f853281aadf067187 P 6bc7d6db2c3b42b7b1dc3f73beab063f: Bootstrap starting.
I20260812 07:03:36.491930 7591 tablet_bootstrap.cc:492] T da92cdb1e42f4d4f853281aadf067187 P 39ca7c5a5e4646e694c94effb0a7c314: No bootstrap required, opened a new log
I20260812 07:03:36.492292 7591 ts_tablet_manager.cc:1397] T da92cdb1e42f4d4f853281aadf067187 P 39ca7c5a5e4646e694c94effb0a7c314: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260812 07:03:36.493649 7593 tablet_bootstrap.cc:654] T da92cdb1e42f4d4f853281aadf067187 P 6bc7d6db2c3b42b7b1dc3f73beab063f: Neither blocks nor log segments found. Creating new log.
I20260812 07:03:36.494958 7593 log.cc:824] T da92cdb1e42f4d4f853281aadf067187 P 6bc7d6db2c3b42b7b1dc3f73beab063f: Log is configured to *not* fsync() on all Append() calls
I20260812 07:03:36.496197 7592 tablet_bootstrap.cc:492] T da92cdb1e42f4d4f853281aadf067187 P 0f00c0808cfa45e799c0974c7cd2196f: Bootstrap starting.
I20260812 07:03:36.498761 7593 tablet_bootstrap.cc:492] T da92cdb1e42f4d4f853281aadf067187 P 6bc7d6db2c3b42b7b1dc3f73beab063f: No bootstrap required, opened a new log
I20260812 07:03:36.499079 7593 ts_tablet_manager.cc:1397] T da92cdb1e42f4d4f853281aadf067187 P 6bc7d6db2c3b42b7b1dc3f73beab063f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260812 07:03:36.499737 7592 tablet_bootstrap.cc:654] T da92cdb1e42f4d4f853281aadf067187 P 0f00c0808cfa45e799c0974c7cd2196f: Neither blocks nor log segments found. Creating new log.
I20260812 07:03:36.500979 7592 log.cc:824] T da92cdb1e42f4d4f853281aadf067187 P 0f00c0808cfa45e799c0974c7cd2196f: Log is configured to *not* fsync() on all Append() calls
I20260812 07:03:36.503201 7591 raft_consensus.cc:348] T da92cdb1e42f4d4f853281aadf067187 P 39ca7c5a5e4646e694c94effb0a7c314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f00c0808cfa45e799c0974c7cd2196f" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 34387 } } peers { permanent_uuid: "39ca7c5a5e4646e694c94effb0a7c314" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 37607 } } peers { permanent_uuid: "6bc7d6db2c3b42b7b1dc3f73beab063f" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 37361 } }
I20260812 07:03:36.503665 7591 raft_consensus.cc:374] T da92cdb1e42f4d4f853281aadf067187 P 39ca7c5a5e4646e694c94effb0a7c314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:03:36.503808 7591 raft_consensus.cc:729] T da92cdb1e42f4d4f853281aadf067187 P 39ca7c5a5e4646e694c94effb0a7c314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39ca7c5a5e4646e694c94effb0a7c314, State: Initialized, Role: FOLLOWER
I20260812 07:03:36.504292 7592 tablet_bootstrap.cc:492] T da92cdb1e42f4d4f853281aadf067187 P 0f00c0808cfa45e799c0974c7cd2196f: No bootstrap required, opened a new log
I20260812 07:03:36.504571 7592 ts_tablet_manager.cc:1397] T da92cdb1e42f4d4f853281aadf067187 P 0f00c0808cfa45e799c0974c7cd2196f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260812 07:03:36.504442 7591 consensus_queue.cc:260] T da92cdb1e42f4d4f853281aadf067187 P 39ca7c5a5e4646e694c94effb0a7c314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f00c0808cfa45e799c0974c7cd2196f" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 34387 } } peers { permanent_uuid: "39ca7c5a5e4646e694c94effb0a7c314" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 37607 } } peers { permanent_uuid: "6bc7d6db2c3b42b7b1dc3f73beab063f" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 37361 } }
I20260812 07:03:36.506294 7591 ts_tablet_manager.cc:1428] T da92cdb1e42f4d4f853281aadf067187 P 39ca7c5a5e4646e694c94effb0a7c314: Time spent starting tablet: real 0.014s user 0.003s sys 0.011s
I20260812 07:03:36.508572 7593 raft_consensus.cc:348] T da92cdb1e42f4d4f853281aadf067187 P 6bc7d6db2c3b42b7b1dc3f73beab063f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f00c0808cfa45e799c0974c7cd2196f" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 34387 } } peers { permanent_uuid: "39ca7c5a5e4646e694c94effb0a7c314" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 37607 } } peers { permanent_uuid: "6bc7d6db2c3b42b7b1dc3f73beab063f" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 37361 } }
I20260812 07:03:36.509040 7593 raft_consensus.cc:374] T da92cdb1e42f4d4f853281aadf067187 P 6bc7d6db2c3b42b7b1dc3f73beab063f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:03:36.509156 7593 raft_consensus.cc:729] T da92cdb1e42f4d4f853281aadf067187 P 6bc7d6db2c3b42b7b1dc3f73beab063f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bc7d6db2c3b42b7b1dc3f73beab063f, State: Initialized, Role: FOLLOWER
I20260812 07:03:36.509790 7593 consensus_queue.cc:260] T da92cdb1e42f4d4f853281aadf067187 P 6bc7d6db2c3b42b7b1dc3f73beab063f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f00c0808cfa45e799c0974c7cd2196f" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 34387 } } peers { permanent_uuid: "39ca7c5a5e4646e694c94effb0a7c314" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 37607 } } peers { permanent_uuid: "6bc7d6db2c3b42b7b1dc3f73beab063f" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 37361 } }
I20260812 07:03:36.510998 7411 heartbeater.cc:503] Master 127.6.61.62:37143 was elected leader, sending a full tablet report...
I20260812 07:03:36.511775 7593 ts_tablet_manager.cc:1428] T da92cdb1e42f4d4f853281aadf067187 P 6bc7d6db2c3b42b7b1dc3f73beab063f: Time spent starting tablet: real 0.013s user 0.006s sys 0.006s
I20260812 07:03:36.514590 7592 raft_consensus.cc:348] T da92cdb1e42f4d4f853281aadf067187 P 0f00c0808cfa45e799c0974c7cd2196f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f00c0808cfa45e799c0974c7cd2196f" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 34387 } } peers { permanent_uuid: "39ca7c5a5e4646e694c94effb0a7c314" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 37607 } } peers { permanent_uuid: "6bc7d6db2c3b42b7b1dc3f73beab063f" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 37361 } }
I20260812 07:03:36.515051 7592 raft_consensus.cc:374] T da92cdb1e42f4d4f853281aadf067187 P 0f00c0808cfa45e799c0974c7cd2196f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:03:36.515182 7592 raft_consensus.cc:729] T da92cdb1e42f4d4f853281aadf067187 P 0f00c0808cfa45e799c0974c7cd2196f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f00c0808cfa45e799c0974c7cd2196f, State: Initialized, Role: FOLLOWER
I20260812 07:03:36.515820 7592 consensus_queue.cc:260] T da92cdb1e42f4d4f853281aadf067187 P 0f00c0808cfa45e799c0974c7cd2196f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f00c0808cfa45e799c0974c7cd2196f" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 34387 } } peers { permanent_uuid: "39ca7c5a5e4646e694c94effb0a7c314" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 37607 } } peers { permanent_uuid: "6bc7d6db2c3b42b7b1dc3f73beab063f" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 37361 } }
I20260812 07:03:36.516798 7569 heartbeater.cc:503] Master 127.6.61.62:37143 was elected leader, sending a full tablet report...
I20260812 07:03:36.517409 7592 ts_tablet_manager.cc:1428] T da92cdb1e42f4d4f853281aadf067187 P 0f00c0808cfa45e799c0974c7cd2196f: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
W20260812 07:03:36.561570 7261 tablet.cc:2369] T da92cdb1e42f4d4f853281aadf067187 P 39ca7c5a5e4646e694c94effb0a7c314: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 07:03:36.579787 7412 tablet.cc:2369] T da92cdb1e42f4d4f853281aadf067187 P 6bc7d6db2c3b42b7b1dc3f73beab063f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 07:03:36.617028 7570 tablet.cc:2369] T da92cdb1e42f4d4f853281aadf067187 P 0f00c0808cfa45e799c0974c7cd2196f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:03:36.641708 7601 raft_consensus.cc:482] T da92cdb1e42f4d4f853281aadf067187 P 0f00c0808cfa45e799c0974c7cd2196f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:03:36.642145 7601 raft_consensus.cc:504] T da92cdb1e42f4d4f853281aadf067187 P 0f00c0808cfa45e799c0974c7cd2196f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f00c0808cfa45e799c0974c7cd2196f" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 34387 } } peers { permanent_uuid: "39ca7c5a5e4646e694c94effb0a7c314" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 37607 } } peers { permanent_uuid: "6bc7d6db2c3b42b7b1dc3f73beab063f" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 37361 } }
I20260812 07:03:36.644109 7601 leader_election.cc:288] T da92cdb1e42f4d4f853281aadf067187 P 0f00c0808cfa45e799c0974c7cd2196f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39ca7c5a5e4646e694c94effb0a7c314 (127.6.61.1:37607), 6bc7d6db2c3b42b7b1dc3f73beab063f (127.6.61.2:37361)
I20260812 07:03:36.650677 7587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:36.644851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:45315 (local address 127.6.61.2:37361)
0812 07:03:36.645060 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:36.645065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:36.645081 (+ 16us) server_negotiation.cc:407] Connection header received
0812 07:03:36.645144 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:36.645147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:36.645238 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:36.645357 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:03:36.645875 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.646746 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.648480 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.649743 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.649924 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:36.650046 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:36.650359 (+ 313us) server_negotiation.cc:299] Negotiation successful
0812 07:03:36.650450 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 07:03:36.650766 7573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:36.644383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:37361 (local address 127.6.61.3:45315)
0812 07:03:36.644695 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:36.644722 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:36.644861 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:36.645387 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:36.645392 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:36.645428 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:03:36.645692 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:36.645702 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.646870 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.646878 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:36.648293 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:36.648310 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.649941 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.649964 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:36.650162 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:36.650200 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 07:03:36.650339 (+ 139us) client_negotiation.cc:240] Negotiation successful
0812 07:03:36.650487 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260812 07:03:36.652205 7602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:36.644501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:37607 (local address 127.6.61.3:41781)
0812 07:03:36.645643 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:36.645664 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:36.645886 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:36.646279 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:36.646283 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:36.646307 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:03:36.646679 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:36.646688 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.647795 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.647800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:36.649877 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:36.649892 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.651456 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.651465 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:36.651607 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:36.651631 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 07:03:36.651714 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 07:03:36.651806 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1003,"thread_start_us":618,"threads_started":1}
I20260812 07:03:36.652215 7590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:36.644670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:41781 (local address 127.6.61.1:37607)
0812 07:03:36.644902 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:36.644910 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:36.645770 (+ 860us) server_negotiation.cc:407] Connection header received
0812 07:03:36.646068 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:36.646072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:36.646145 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:36.646263 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:03:36.646834 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.647642 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.650030 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.651259 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.651360 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:36.651430 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:36.651791 (+ 361us) server_negotiation.cc:299] Negotiation successful
0812 07:03:36.651940 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 07:03:36.652045 7361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da92cdb1e42f4d4f853281aadf067187" candidate_uuid: "0f00c0808cfa45e799c0974c7cd2196f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bc7d6db2c3b42b7b1dc3f73beab063f" is_pre_election: true
I20260812 07:03:36.652707 7361 raft_consensus.cc:2454] T da92cdb1e42f4d4f853281aadf067187 P 6bc7d6db2c3b42b7b1dc3f73beab063f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f00c0808cfa45e799c0974c7cd2196f in term 0.
I20260812 07:03:36.653466 7440 leader_election.cc:302] T da92cdb1e42f4d4f853281aadf067187 P 0f00c0808cfa45e799c0974c7cd2196f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f00c0808cfa45e799c0974c7cd2196f, 6bc7d6db2c3b42b7b1dc3f73beab063f; no voters:
I20260812 07:03:36.653456 7207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da92cdb1e42f4d4f853281aadf067187" candidate_uuid: "0f00c0808cfa45e799c0974c7cd2196f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39ca7c5a5e4646e694c94effb0a7c314" is_pre_election: true
I20260812 07:03:36.653883 7601 raft_consensus.cc:2781] T da92cdb1e42f4d4f853281aadf067187 P 0f00c0808cfa45e799c0974c7cd2196f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:03:36.654021 7601 raft_consensus.cc:482] T da92cdb1e42f4d4f853281aadf067187 P 0f00c0808cfa45e799c0974c7cd2196f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:03:36.654119 7601 raft_consensus.cc:3037] T da92cdb1e42f4d4f853281aadf067187 P 0f00c0808cfa45e799c0974c7cd2196f [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:03:36.654040 7207 raft_consensus.cc:2454] T da92cdb1e42f4d4f853281aadf067187 P 39ca7c5a5e4646e694c94effb0a7c314 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f00c0808cfa45e799c0974c7cd2196f in term 0.
I20260812 07:03:36.656641 7601 raft_consensus.cc:504] T da92cdb1e42f4d4f853281aadf067187 P 0f00c0808cfa45e799c0974c7cd2196f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f00c0808cfa45e799c0974c7cd2196f" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 34387 } } peers { permanent_uuid: "39ca7c5a5e4646e694c94effb0a7c314" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 37607 } } peers { permanent_uuid: "6bc7d6db2c3b42b7b1dc3f73beab063f" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 37361 } }
I20260812 07:03:36.657312 7601 leader_election.cc:288] T da92cdb1e42f4d4f853281aadf067187 P 0f00c0808cfa45e799c0974c7cd2196f [CANDIDATE]: Term 1 election: Requested vote from peers 39ca7c5a5e4646e694c94effb0a7c314 (127.6.61.1:37607), 6bc7d6db2c3b42b7b1dc3f73beab063f (127.6.61.2:37361)
I20260812 07:03:36.657843 7207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da92cdb1e42f4d4f853281aadf067187" candidate_uuid: "0f00c0808cfa45e799c0974c7cd2196f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39ca7c5a5e4646e694c94effb0a7c314"
I20260812 07:03:36.657909 7361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da92cdb1e42f4d4f853281aadf067187" candidate_uuid: "0f00c0808cfa45e799c0974c7cd2196f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bc7d6db2c3b42b7b1dc3f73beab063f"
I20260812 07:03:36.658103 7207 raft_consensus.cc:3037] T da92cdb1e42f4d4f853281aadf067187 P 39ca7c5a5e4646e694c94effb0a7c314 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:03:36.658126 7361 raft_consensus.cc:3037] T da92cdb1e42f4d4f853281aadf067187 P 6bc7d6db2c3b42b7b1dc3f73beab063f [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:03:36.661496 7207 raft_consensus.cc:2454] T da92cdb1e42f4d4f853281aadf067187 P 39ca7c5a5e4646e694c94effb0a7c314 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f00c0808cfa45e799c0974c7cd2196f in term 1.
I20260812 07:03:36.661496 7361 raft_consensus.cc:2454] T da92cdb1e42f4d4f853281aadf067187 P 6bc7d6db2c3b42b7b1dc3f73beab063f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f00c0808cfa45e799c0974c7cd2196f in term 1.
I20260812 07:03:36.661970 7440 leader_election.cc:302] T da92cdb1e42f4d4f853281aadf067187 P 0f00c0808cfa45e799c0974c7cd2196f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f00c0808cfa45e799c0974c7cd2196f, 6bc7d6db2c3b42b7b1dc3f73beab063f; no voters:
I20260812 07:03:36.662300 7601 raft_consensus.cc:2781] T da92cdb1e42f4d4f853281aadf067187 P 0f00c0808cfa45e799c0974c7cd2196f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:03:36.662779 7601 raft_consensus.cc:686] T da92cdb1e42f4d4f853281aadf067187 P 0f00c0808cfa45e799c0974c7cd2196f [term 1 LEADER]: Becoming Leader. State: Replica: 0f00c0808cfa45e799c0974c7cd2196f, State: Running, Role: LEADER
I20260812 07:03:36.663302 7601 consensus_queue.cc:237] T da92cdb1e42f4d4f853281aadf067187 P 0f00c0808cfa45e799c0974c7cd2196f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f00c0808cfa45e799c0974c7cd2196f" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 34387 } } peers { permanent_uuid: "39ca7c5a5e4646e694c94effb0a7c314" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 37607 } } peers { permanent_uuid: "6bc7d6db2c3b42b7b1dc3f73beab063f" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 37361 } }
I20260812 07:03:36.668257 7037 catalog_manager.cc:5495] T da92cdb1e42f4d4f853281aadf067187 P 0f00c0808cfa45e799c0974c7cd2196f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f00c0808cfa45e799c0974c7cd2196f (127.6.61.3). New cstate: current_term: 1 leader_uuid: "0f00c0808cfa45e799c0974c7cd2196f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f00c0808cfa45e799c0974c7cd2196f" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 34387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39ca7c5a5e4646e694c94effb0a7c314" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 37607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bc7d6db2c3b42b7b1dc3f73beab063f" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 37361 } health_report { overall_health: UNKNOWN } } }
Aug 12 07:03:36 dist-test-slave-69rg krb5kdc[6986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.1@KRBTEST.COM
I20260812 07:03:36.697585 7590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:36.681664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54056 (local address 127.6.61.1:37607)
0812 07:03:36.681907 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:36.681912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:36.682500 (+ 588us) server_negotiation.cc:407] Connection header received
0812 07:03:36.682716 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:36.682722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:36.682823 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:36.682936 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:36.684512 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.685318 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.686697 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.687145 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.687293 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:36.691521 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:36.691537 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:36.691554 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:36.691590 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:36.694650 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:36.695380 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:36.695387 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:36.695390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:36.695526 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:36.695983 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:36.695988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:36.695997 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:36.696351 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:36.696689 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:36.696970 (+ 281us) server_negotiation.cc:299] Negotiation successful
0812 07:03:36.697207 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Aug 12 07:03:36 dist-test-slave-69rg krb5kdc[6986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518216, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 07:03:36 dist-test-slave-69rg krb5kdc[6986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518216, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.61.1@KRBTEST.COM
I20260812 07:03:36.756939 7590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:36.739950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54068 (local address 127.6.61.1:37607)
0812 07:03:36.740171 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:36.740181 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:36.740594 (+ 413us) server_negotiation.cc:407] Connection header received
0812 07:03:36.740827 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:36.740831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:36.740904 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:36.741020 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:36.742579 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.743358 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.744695 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.745267 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.745406 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:36.749695 (+ 4289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:36.749702 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:36.749709 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:36.749765 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:36.753652 (+ 3887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:36.754509 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:36.754516 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:36.754520 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:36.754696 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:36.755210 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:36.755216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:36.755219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:36.755657 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:36.755994 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:36.756288 (+ 294us) server_negotiation.cc:299] Negotiation successful
0812 07:03:36.756547 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Aug 12 07:03:36 dist-test-slave-69rg krb5kdc[6986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 07:03:36 dist-test-slave-69rg krb5kdc[6986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.1@KRBTEST.COM
I20260812 07:03:36.813933 7590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:36.798501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54082 (local address 127.6.61.1:37607)
0812 07:03:36.798735 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:36.798745 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:36.799095 (+ 350us) server_negotiation.cc:407] Connection header received
0812 07:03:36.799343 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:36.799348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:36.799417 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:36.799536 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:36.801149 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.801944 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.803169 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.803601 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.803739 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:36.808068 (+ 4329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:36.808074 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:36.808078 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:36.808114 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:36.811061 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:36.811766 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:36.811770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:36.811773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:36.811896 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:36.812232 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:36.812237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:36.812240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:36.812621 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:36.812882 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:36.813409 (+ 527us) server_negotiation.cc:299] Negotiation successful
0812 07:03:36.813593 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260812 07:03:36.815718 7162 tablet_service.cc:2908] Rejecting scan request for tablet da92cdb1e42f4d4f853281aadf067187: Uninitialized: clean time has not yet been initialized
I20260812 07:03:36.819510 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 7111
I20260812 07:03:36.833766 7590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:36.820369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54090 (local address 127.6.61.1:37607)
0812 07:03:36.820531 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:36.820535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:36.820888 (+ 353us) server_negotiation.cc:407] Connection header received
0812 07:03:36.821051 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:36.821055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:36.821117 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:36.821206 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:36.822601 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.823289 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.824542 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.824954 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.825158 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:36.827509 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:36.827515 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:36.827519 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:36.827559 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:36.831081 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:36.831759 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:36.831765 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:36.831769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:36.831911 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:36.832227 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:36.832232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:36.832236 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:36.832578 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:36.832854 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:36.833165 (+ 311us) server_negotiation.cc:299] Negotiation successful
0812 07:03:36.833417 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 07:03:36.834486 7255 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:03:36.929948 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 7111
I20260812 07:03:36.951401 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 7268
Aug 12 07:03:36 dist-test-slave-69rg krb5kdc[6986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.2@KRBTEST.COM
I20260812 07:03:36.966960 7587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:36.952409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58376 (local address 127.6.61.2:37361)
0812 07:03:36.952605 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:36.952610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:36.952628 (+ 18us) server_negotiation.cc:407] Connection header received
0812 07:03:36.952714 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:36.952718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:36.952785 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:36.952866 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:36.954235 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.955007 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.956490 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:36.956871 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:36.956967 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:36.961213 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:36.961230 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:36.961248 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:36.961286 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:36.964172 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:36.964909 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:36.964916 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:36.964918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:36.965073 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:36.965589 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:36.965596 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:36.965601 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:36.966050 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:36.966376 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:36.966531 (+ 155us) server_negotiation.cc:299] Negotiation successful
0812 07:03:36.966709 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 07:03:36.967723 7405 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:03:37.057514 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 7268
I20260812 07:03:37.079953 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 7416
Aug 12 07:03:37 dist-test-slave-69rg krb5kdc[6986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.3@KRBTEST.COM
I20260812 07:03:37.096149 7626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:37.080803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58662 (local address 127.6.61.3:34387)
0812 07:03:37.081352 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:37.081358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:37.081373 (+ 15us) server_negotiation.cc:407] Connection header received
0812 07:03:37.081431 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:37.081434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:37.081495 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:37.081631 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:37.083018 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:37.083876 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:37.085173 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:37.085585 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:37.085727 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:37.090247 (+ 4520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:37.090265 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:37.090285 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:37.090321 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:37.093296 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:37.094145 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:37.094158 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:37.094164 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:37.094346 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:37.094765 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:37.094770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:37.094772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:37.095122 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:37.095359 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:37.095652 (+ 293us) server_negotiation.cc:299] Negotiation successful
0812 07:03:37.095860 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":282,"threads_started":1}
I20260812 07:03:37.096880 7564 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:03:37.186021 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 7416
I20260812 07:03:37.206895 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 7002
I20260812 07:03:37.208033 7076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:03:37.318053 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 7002
2026-08-12T07:03:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4428 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:03:37 dist-test-slave-69rg krb5kdc[7634](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:03:37 dist-test-slave-69rg krb5kdc[7634](info): set up 2 sockets
Aug 12 07:03:37 dist-test-slave-69rg krb5kdc[7634](info): commencing operation
krb5kdc: starting...
W20260812 07:03:38.493273 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:03:38 dist-test-slave-69rg krb5kdc[7634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:03:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:03:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
I20260812 07:03:38.706284 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:44745
--webserver_interface=127.6.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:46099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:44745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:03:39.037114 7649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:03:39.037397 7649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:03:39.037487 7649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:03:39.044473 7649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:03:39.044566 7649 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 07:03:39.044606 7649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:03:39.044667 7649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:03:39.044716 7649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:03:39.044771 7649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:03:39.055133 7649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:46099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:44745
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:44745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:03:39.056366 7649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:03:39.057650 7649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:03:39.070001 7656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:39.070029 7655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:39.070053 7658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:03:39.070839 7649 server_base.cc:1034] running on GCE node
I20260812 07:03:39.071666 7649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:03:39.073074 7649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:03:39.074295 7649 hybrid_clock.cc:648] HybridClock initialized: now 1786518219074227 us; error 66 us; skew 500 ppm
I20260812 07:03:39.074625 7649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:03:39 dist-test-slave-69rg krb5kdc[7634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518219, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:03:39.077991 7649 init.cc:377] Logged in from keytab as kudu/127.6.61.62@KRBTEST.COM (short username kudu)
I20260812 07:03:39.079385 7649 webserver.cc:460] Webserver started at http://127.6.61.62:39897/ using document root <none> and password file <none>
I20260812 07:03:39.080039 7649 fs_manager.cc:359] Metadata directory not provided
I20260812 07:03:39.080125 7649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:03:39.080387 7649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:03:39.083333 7649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "2f1dea41f34141ffa6efc53e55582c43"
format_stamp: "Formatted at 2026-08-12 07:03:39 on dist-test-slave-69rg"
server_key: "3f9db85446767cb29baf671b6c89a28a"
server_key_iv: "6f0a0521001a7b96cec326b19bb9172f"
server_key_version: "encryptionkey@0"
I20260812 07:03:39.084084 7649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "2f1dea41f34141ffa6efc53e55582c43"
format_stamp: "Formatted at 2026-08-12 07:03:39 on dist-test-slave-69rg"
server_key: "3f9db85446767cb29baf671b6c89a28a"
server_key_iv: "6f0a0521001a7b96cec326b19bb9172f"
server_key_version: "encryptionkey@0"
I20260812 07:03:39.089068 7649 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 07:03:39.092615 7665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:39.094117 7649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:03:39.094313 7649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "2f1dea41f34141ffa6efc53e55582c43"
format_stamp: "Formatted at 2026-08-12 07:03:39 on dist-test-slave-69rg"
server_key: "3f9db85446767cb29baf671b6c89a28a"
server_key_iv: "6f0a0521001a7b96cec326b19bb9172f"
server_key_version: "encryptionkey@0"
I20260812 07:03:39.094532 7649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:03:39.110589 7649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:03:39.114964 7649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:03:39.115267 7649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:03:39.137480 7649 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:44745
I20260812 07:03:39.137490 7737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:44745 every 8 connection(s)
I20260812 07:03:39.138965 7649 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:03:39.142712 7738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:03:39.146334 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 7649
I20260812 07:03:39.146580 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:03:39.147446 6388 external_mini_cluster.cc:1428] Setting key 15b7927e6c5c5698b1854d3146a388a0
I20260812 07:03:39.156193 7738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f1dea41f34141ffa6efc53e55582c43: Bootstrap starting.
I20260812 07:03:39.160142 7738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f1dea41f34141ffa6efc53e55582c43: Neither blocks nor log segments found. Creating new log.
I20260812 07:03:39.161540 7738 log.cc:824] T 00000000000000000000000000000000 P 2f1dea41f34141ffa6efc53e55582c43: Log is configured to *not* fsync() on all Append() calls
Aug 12 07:03:39 dist-test-slave-69rg krb5kdc[7634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:03:39.164722 7738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f1dea41f34141ffa6efc53e55582c43: No bootstrap required, opened a new log
I20260812 07:03:39.171370 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:39.149537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50072 (local address 127.6.61.62:44745)
0812 07:03:39.150343 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:39.150354 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:39.150397 (+ 43us) server_negotiation.cc:407] Connection header received
0812 07:03:39.151996 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:39.152020 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:39.152402 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:39.152725 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:39.154575 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:39.155883 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:39.157632 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:39.158466 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:39.158605 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:39.163786 (+ 5181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:39.163795 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:39.163810 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:39.163846 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:39.167105 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:39.167876 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:39.167891 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:39.167898 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:39.168073 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:39.168461 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:39.168467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:39.168472 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:39.169015 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:39.169373 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:39.169804 (+ 431us) server_negotiation.cc:299] Negotiation successful
0812 07:03:39.170275 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":402,"threads_started":1}
I20260812 07:03:39.173349 7738 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f1dea41f34141ffa6efc53e55582c43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1dea41f34141ffa6efc53e55582c43" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 44745 } }
I20260812 07:03:39.173751 7738 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f1dea41f34141ffa6efc53e55582c43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:03:39.173928 7738 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f1dea41f34141ffa6efc53e55582c43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f1dea41f34141ffa6efc53e55582c43, State: Initialized, Role: FOLLOWER
I20260812 07:03:39.174606 7738 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f1dea41f34141ffa6efc53e55582c43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1dea41f34141ffa6efc53e55582c43" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 44745 } }
I20260812 07:03:39.174829 7738 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f1dea41f34141ffa6efc53e55582c43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:03:39.174925 7738 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f1dea41f34141ffa6efc53e55582c43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:03:39.175055 7738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f1dea41f34141ffa6efc53e55582c43 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:03:39.177194 7738 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f1dea41f34141ffa6efc53e55582c43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1dea41f34141ffa6efc53e55582c43" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 44745 } }
I20260812 07:03:39.177642 7738 leader_election.cc:302] T 00000000000000000000000000000000 P 2f1dea41f34141ffa6efc53e55582c43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f1dea41f34141ffa6efc53e55582c43; no voters:
I20260812 07:03:39.178254 7738 leader_election.cc:288] T 00000000000000000000000000000000 P 2f1dea41f34141ffa6efc53e55582c43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:03:39.178406 7743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f1dea41f34141ffa6efc53e55582c43 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:03:39.179570 7743 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f1dea41f34141ffa6efc53e55582c43 [term 1 LEADER]: Becoming Leader. State: Replica: 2f1dea41f34141ffa6efc53e55582c43, State: Running, Role: LEADER
I20260812 07:03:39.180177 7743 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f1dea41f34141ffa6efc53e55582c43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1dea41f34141ffa6efc53e55582c43" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 44745 } }
I20260812 07:03:39.180652 7738 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f1dea41f34141ffa6efc53e55582c43 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:03:39.183283 7745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f1dea41f34141ffa6efc53e55582c43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f1dea41f34141ffa6efc53e55582c43. Latest consensus state: current_term: 1 leader_uuid: "2f1dea41f34141ffa6efc53e55582c43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1dea41f34141ffa6efc53e55582c43" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 44745 } } }
I20260812 07:03:39.184670 7744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f1dea41f34141ffa6efc53e55582c43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f1dea41f34141ffa6efc53e55582c43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1dea41f34141ffa6efc53e55582c43" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 44745 } } }
I20260812 07:03:39.184866 7745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f1dea41f34141ffa6efc53e55582c43 [sys.catalog]: This master's current role is: LEADER
I20260812 07:03:39.184950 7744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f1dea41f34141ffa6efc53e55582c43 [sys.catalog]: This master's current role is: LEADER
I20260812 07:03:39.187027 7754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:03:39.192155 7754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:03:39.201210 7754 catalog_manager.cc:1347] Generated new cluster ID: 038dc3970ea94f99b3e4fe53c722ed3a
I20260812 07:03:39.201339 7754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:03:39.234496 7754 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:03:39.236047 7754 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:03:39.251410 7754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f1dea41f34141ffa6efc53e55582c43: Generated new TSK 0
I20260812 07:03:39.252553 7754 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
I20260812 07:03:39.321548 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.1:0
--local_ip_for_outbound_sockets=127.6.61.1
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:44745
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:46099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:03:39.667877 7769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:03:39.668167 7769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:03:39.668339 7769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:03:39.675680 7769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:03:39.675789 7769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:03:39.675973 7769 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 07:03:39.676085 7769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.1
I20260812 07:03:39.686210 7769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:46099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.61.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.61.62:44745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.1
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:03:39.687512 7769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:03:39.688845 7769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:03:39.702325 7779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:39.702404 7786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:39.702404 7782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:03:39.703677 7769 server_base.cc:1034] running on GCE node
I20260812 07:03:39.704278 7769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:03:39.705241 7769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:03:39.706494 7769 hybrid_clock.cc:648] HybridClock initialized: now 1786518219706414 us; error 56 us; skew 500 ppm
I20260812 07:03:39.706799 7769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:03:39 dist-test-slave-69rg krb5kdc[7634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518219, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:03:39.715706 7769 init.cc:377] Logged in from keytab as kudu/127.6.61.1@KRBTEST.COM (short username kudu)
I20260812 07:03:39.717232 7769 webserver.cc:460] Webserver started at http://127.6.61.1:39253/ using document root <none> and password file <none>
I20260812 07:03:39.718003 7769 fs_manager.cc:359] Metadata directory not provided
I20260812 07:03:39.718152 7769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:03:39.718501 7769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:03:39.721437 7769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-0/data/instance:
uuid: "5e52441d04c64aa4bdcf862cdfbf23c9"
format_stamp: "Formatted at 2026-08-12 07:03:39 on dist-test-slave-69rg"
server_key: "56d6da08b7f2305004ea3831c48a4e1f"
server_key_iv: "a2a1c5af9b4a2270e011586265bf90f4"
server_key_version: "encryptionkey@0"
I20260812 07:03:39.722283 7769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance:
uuid: "5e52441d04c64aa4bdcf862cdfbf23c9"
format_stamp: "Formatted at 2026-08-12 07:03:39 on dist-test-slave-69rg"
server_key: "56d6da08b7f2305004ea3831c48a4e1f"
server_key_iv: "a2a1c5af9b4a2270e011586265bf90f4"
server_key_version: "encryptionkey@0"
I20260812 07:03:39.727402 7769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 07:03:39.730780 7795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:39.732095 7769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 07:03:39.732285 7769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-0/wal
uuid: "5e52441d04c64aa4bdcf862cdfbf23c9"
format_stamp: "Formatted at 2026-08-12 07:03:39 on dist-test-slave-69rg"
server_key: "56d6da08b7f2305004ea3831c48a4e1f"
server_key_iv: "a2a1c5af9b4a2270e011586265bf90f4"
server_key_version: "encryptionkey@0"
I20260812 07:03:39.732524 7769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:03:39.751772 7769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:03:39.756358 7769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:03:39.756656 7769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:03:39.759655 7769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:03:39.759773 7769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:39.759932 7769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:03:39.760011 7769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 07:03:39 dist-test-slave-69rg krb5kdc[7634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518219, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:03:39.788254 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:39.760412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:33613 (local address 127.6.61.62:44745)
0812 07:03:39.760986 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:39.760994 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:39.762616 (+ 1622us) server_negotiation.cc:407] Connection header received
0812 07:03:39.765436 (+ 2820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:39.765442 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:39.765542 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:39.765694 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:39.768613 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:39.769609 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:39.773126 (+ 3517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:39.773627 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:39.773746 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:39.779387 (+ 5641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:39.779399 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:39.779408 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:39.779470 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:39.783511 (+ 4041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:39.784702 (+ 1191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:39.784711 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:39.784716 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:39.784885 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:39.785495 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:39.785498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:39.785500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:39.786000 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:39.786171 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:39.787359 (+ 1188us) server_negotiation.cc:299] Negotiation successful
0812 07:03:39.787700 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":310,"threads_started":1}
I20260812 07:03:39.788923 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:39.761064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:44745 (local address 127.6.61.1:33613)
0812 07:03:39.762054 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:39.762110 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:39.765054 (+ 2944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:39.766072 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:39.766089 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:39.767099 (+ 1010us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:03:39.768336 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:39.768375 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:39.769845 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:39.769854 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:39.772920 (+ 3066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:39.772944 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:39.773961 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:39.773974 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:39.774387 (+ 413us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:39.775308 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:03:39.775352 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:03:39.779052 (+ 3700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:03:39.783720 (+ 4668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:39.783728 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:39.783742 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:39.784486 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:39.785053 (+ 567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:39.785057 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:39.785060 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:39.785364 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:39.786144 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:03:39.786152 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:03:39.786782 (+ 630us) client_negotiation.cc:769] Sending connection context
0812 07:03:39.787187 (+ 405us) client_negotiation.cc:240] Negotiation successful
0812 07:03:39.787656 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":381,"threads_started":1}
I20260812 07:03:39.813259 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:39.795538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:33873 (local address 127.6.61.62:44745)
0812 07:03:39.795747 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:39.795755 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:39.796588 (+ 833us) server_negotiation.cc:407] Connection header received
0812 07:03:39.796798 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:39.796804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:39.796868 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:39.796948 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:39.798646 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:39.799942 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:39.802621 (+ 2679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:39.803288 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:39.803430 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:39.806534 (+ 3104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:39.806543 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:39.806551 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:39.806602 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:39.810295 (+ 3693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:39.811286 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:39.811290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:39.811292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:39.811423 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:39.811833 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:39.811836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:39.811838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:39.812084 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:39.812239 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:39.812668 (+ 429us) server_negotiation.cc:299] Negotiation successful
0812 07:03:39.812897 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 07:03:39.813374 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:39.795613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:44745 (local address 127.6.61.1:33873)
0812 07:03:39.796453 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:39.796478 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:39.796659 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:39.797019 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:39.797027 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:39.797852 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:03:39.798416 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:39.798436 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:39.800175 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:39.800184 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:39.802407 (+ 2223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:39.802433 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:39.803748 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:39.803755 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:39.803946 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:39.804666 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:03:39.804705 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:03:39.806264 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:03:39.810435 (+ 4171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:39.810443 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:39.810449 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:39.811153 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:39.811534 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:39.811539 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:39.811542 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:39.811715 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:39.812211 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:03:39.812216 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:03:39.812402 (+ 186us) client_negotiation.cc:769] Sending connection context
0812 07:03:39.812542 (+ 140us) client_negotiation.cc:240] Negotiation successful
0812 07:03:39.812883 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":461,"threads_started":1}
I20260812 07:03:39.815006 7769 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.1:38735
I20260812 07:03:39.815101 7935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.1:38735 every 8 connection(s)
I20260812 07:03:39.816653 7769 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
I20260812 07:03:39.819022 7936 heartbeater.cc:347] Connected to a master server at 127.6.61.62:44745
I20260812 07:03:39.819260 7936 heartbeater.cc:464] Registering TS with master...
I20260812 07:03:39.819903 7936 heartbeater.cc:511] Master 127.6.61.62:44745 requested a full tablet report, sending...
I20260812 07:03:39.821897 7683 ts_manager.cc:194] Registered new tserver with Master: 5e52441d04c64aa4bdcf862cdfbf23c9 (127.6.61.1:38735)
I20260812 07:03:39.823402 7683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.1@KRBTEST.COM'} at 127.6.61.1:33613
I20260812 07:03:39.824160 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 7769
I20260812 07:03:39.824405 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance
I20260812 07:03:39.825110 6388 external_mini_cluster.cc:1428] Setting key 7cfcf0229dd81a7a2ec0121beea06435
WARNING: no policy specified for kudu/127.6.61.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
I20260812 07:03:39.869599 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.2:0
--local_ip_for_outbound_sockets=127.6.61.2
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:44745
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:46099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:03:40.211966 7940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:03:40.212309 7940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:03:40.212474 7940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:03:40.219995 7940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:03:40.220163 7940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:03:40.220311 7940 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 07:03:40.220455 7940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.2
I20260812 07:03:40.231024 7940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:46099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.61.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.61.62:44745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.2
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:03:40.232411 7940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:03:40.233759 7940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:03:40.247196 7948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:40.247228 7949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:40.247475 7951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:03:40.247967 7940 server_base.cc:1034] running on GCE node
I20260812 07:03:40.248587 7940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:03:40.249531 7940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:03:40.250775 7940 hybrid_clock.cc:648] HybridClock initialized: now 1786518220250692 us; error 60 us; skew 500 ppm
I20260812 07:03:40.251091 7940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:03:40 dist-test-slave-69rg krb5kdc[7634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518220, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:03:40.254570 7940 init.cc:377] Logged in from keytab as kudu/127.6.61.2@KRBTEST.COM (short username kudu)
I20260812 07:03:40.256022 7940 webserver.cc:460] Webserver started at http://127.6.61.2:39061/ using document root <none> and password file <none>
I20260812 07:03:40.256748 7940 fs_manager.cc:359] Metadata directory not provided
I20260812 07:03:40.256891 7940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:03:40.257239 7940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:03:40.260264 7940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-1/data/instance:
uuid: "c335f4241d1f4edcb6a58d348f3e7736"
format_stamp: "Formatted at 2026-08-12 07:03:40 on dist-test-slave-69rg"
server_key: "c28a62bee09c758f9c1ffd5536cbfb41"
server_key_iv: "512f23225416026106ad0f23aaa97e8f"
server_key_version: "encryptionkey@0"
I20260812 07:03:40.261093 7940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance:
uuid: "c335f4241d1f4edcb6a58d348f3e7736"
format_stamp: "Formatted at 2026-08-12 07:03:40 on dist-test-slave-69rg"
server_key: "c28a62bee09c758f9c1ffd5536cbfb41"
server_key_iv: "512f23225416026106ad0f23aaa97e8f"
server_key_version: "encryptionkey@0"
I20260812 07:03:40.266573 7940 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260812 07:03:40.270126 7959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:40.271478 7940 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:03:40.271661 7940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-1/wal
uuid: "c335f4241d1f4edcb6a58d348f3e7736"
format_stamp: "Formatted at 2026-08-12 07:03:40 on dist-test-slave-69rg"
server_key: "c28a62bee09c758f9c1ffd5536cbfb41"
server_key_iv: "512f23225416026106ad0f23aaa97e8f"
server_key_version: "encryptionkey@0"
I20260812 07:03:40.271896 7940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:03:40.293601 7940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:03:40.297737 7940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:03:40.298066 7940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:03:40.300776 7940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:03:40.300889 7940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:40.301052 7940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:03:40.301131 7940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 07:03:40 dist-test-slave-69rg krb5kdc[7634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518220, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:03:40.326066 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:40.301478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:33373 (local address 127.6.61.62:44745)
0812 07:03:40.301787 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:40.301795 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:40.303238 (+ 1443us) server_negotiation.cc:407] Connection header received
0812 07:03:40.305248 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:40.305254 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:40.305360 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:40.305513 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:40.307928 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:40.309014 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:40.311721 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:40.312131 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:40.312270 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:40.317872 (+ 5602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:40.317881 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:40.317889 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:40.317937 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:40.321473 (+ 3536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:40.322690 (+ 1217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:40.322698 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:40.322703 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:40.322871 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:40.323511 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:40.323518 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:40.323523 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:40.323977 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:40.324174 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:40.325172 (+ 998us) server_negotiation.cc:299] Negotiation successful
0812 07:03:40.325497 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 07:03:40.326759 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:40.302145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:44745 (local address 127.6.61.2:33373)
0812 07:03:40.303032 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:40.303090 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:40.304951 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:40.305795 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:40.305809 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:40.306640 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:03:40.307734 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:40.307767 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:40.309172 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:40.309178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:40.311555 (+ 2377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:40.311568 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:40.312399 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:40.312411 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:40.312720 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:40.313492 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:03:40.313542 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:03:40.317536 (+ 3994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:03:40.321634 (+ 4098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:40.321647 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:40.321671 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:40.322518 (+ 847us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:40.323033 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:40.323037 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:40.323041 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:40.323346 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:40.324142 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:03:40.324153 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:03:40.324618 (+ 465us) client_negotiation.cc:769] Sending connection context
0812 07:03:40.325007 (+ 389us) client_negotiation.cc:240] Negotiation successful
0812 07:03:40.325405 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":378,"threads_started":1}
I20260812 07:03:40.350390 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:40.333490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:36449 (local address 127.6.61.62:44745)
0812 07:03:40.333732 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:40.333741 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:40.334289 (+ 548us) server_negotiation.cc:407] Connection header received
0812 07:03:40.334430 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:40.334438 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:40.334558 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:40.334688 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:40.335564 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:40.336917 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:40.339705 (+ 2788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:40.340246 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:40.340395 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:40.343014 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:40.343022 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:40.343030 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:40.343077 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:40.346470 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:40.347583 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:40.347592 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:40.347597 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:40.347786 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:40.348254 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:40.348258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:40.348260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:40.348583 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:40.348700 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:40.350005 (+ 1305us) server_negotiation.cc:299] Negotiation successful
0812 07:03:40.350171 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 07:03:40.350451 8053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:40.333488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:44745 (local address 127.6.61.2:36449)
0812 07:03:40.334192 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:40.334208 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:40.334313 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:40.334699 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:40.334702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:40.335148 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:03:40.335423 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:40.335431 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:40.337129 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:40.337137 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:40.339502 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:40.339524 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:40.340483 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:40.340489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:40.340659 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:40.341319 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:03:40.341343 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:03:40.342795 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:03:40.346622 (+ 3827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:40.346627 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:40.346635 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:40.347388 (+ 753us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:40.347940 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:40.347948 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:40.347954 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:40.348142 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:40.348728 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:03:40.348736 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:03:40.349077 (+ 341us) client_negotiation.cc:769] Sending connection context
0812 07:03:40.349280 (+ 203us) client_negotiation.cc:240] Negotiation successful
0812 07:03:40.349757 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":387,"threads_started":1}
I20260812 07:03:40.350940 7940 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.2:36919
I20260812 07:03:40.351020 8099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.2:36919 every 8 connection(s)
I20260812 07:03:40.352484 7940 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
I20260812 07:03:40.354279 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 7940
I20260812 07:03:40.354549 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance
I20260812 07:03:40.355149 6388 external_mini_cluster.cc:1428] Setting key e8a04894cab65fa5b635d77f1ce1d16b
I20260812 07:03:40.355242 8101 heartbeater.cc:347] Connected to a master server at 127.6.61.62:44745
I20260812 07:03:40.355477 8101 heartbeater.cc:464] Registering TS with master...
I20260812 07:03:40.356209 8101 heartbeater.cc:511] Master 127.6.61.62:44745 requested a full tablet report, sending...
I20260812 07:03:40.357844 7683 ts_manager.cc:194] Registered new tserver with Master: c335f4241d1f4edcb6a58d348f3e7736 (127.6.61.2:36919)
I20260812 07:03:40.359349 7683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.2@KRBTEST.COM'} at 127.6.61.2:33373
WARNING: no policy specified for kudu/127.6.61.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
I20260812 07:03:40.402252 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.3:0
--local_ip_for_outbound_sockets=127.6.61.3
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:44745
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:46099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:03:40.733228 8105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:03:40.733532 8105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:03:40.733704 8105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:03:40.740909 8105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:03:40.741055 8105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:03:40.741211 8105 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 07:03:40.741350 8105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.3
I20260812 07:03:40.751721 8105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:46099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.61.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.61.62:44745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.3
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:03:40.753046 8105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:03:40.754515 8105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:03:40.768100 8112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:40.768077 8115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:40.768303 8110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:03:40.768844 8105 server_base.cc:1034] running on GCE node
I20260812 07:03:40.769460 8105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:03:40.770444 8105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:03:40.771690 8105 hybrid_clock.cc:648] HybridClock initialized: now 1786518220771628 us; error 39 us; skew 500 ppm
I20260812 07:03:40.771987 8105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:03:40 dist-test-slave-69rg krb5kdc[7634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518220, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:03:40.775336 8105 init.cc:377] Logged in from keytab as kudu/127.6.61.3@KRBTEST.COM (short username kudu)
I20260812 07:03:40.776937 8105 webserver.cc:460] Webserver started at http://127.6.61.3:44733/ using document root <none> and password file <none>
I20260812 07:03:40.777626 8105 fs_manager.cc:359] Metadata directory not provided
I20260812 07:03:40.777768 8105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:03:40.778138 8105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:03:40.781075 8105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-2/data/instance:
uuid: "fb75cd25c1f045a4a036e627b374310a"
format_stamp: "Formatted at 2026-08-12 07:03:40 on dist-test-slave-69rg"
server_key: "df18ef3023df1ab6728589bb50eabcc4"
server_key_iv: "f370c413aff3b6e82ed66f30c9c2699b"
server_key_version: "encryptionkey@0"
I20260812 07:03:40.781944 8105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance:
uuid: "fb75cd25c1f045a4a036e627b374310a"
format_stamp: "Formatted at 2026-08-12 07:03:40 on dist-test-slave-69rg"
server_key: "df18ef3023df1ab6728589bb50eabcc4"
server_key_iv: "f370c413aff3b6e82ed66f30c9c2699b"
server_key_version: "encryptionkey@0"
I20260812 07:03:40.787091 8105 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 07:03:40.790477 8124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:40.791927 8105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:03:40.792133 8105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-2/wal
uuid: "fb75cd25c1f045a4a036e627b374310a"
format_stamp: "Formatted at 2026-08-12 07:03:40 on dist-test-slave-69rg"
server_key: "df18ef3023df1ab6728589bb50eabcc4"
server_key_iv: "f370c413aff3b6e82ed66f30c9c2699b"
server_key_version: "encryptionkey@0"
I20260812 07:03:40.792393 8105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:03:40.822716 8105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:03:40.826342 7936 heartbeater.cc:503] Master 127.6.61.62:44745 was elected leader, sending a full tablet report...
I20260812 07:03:40.826903 8105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:03:40.827198 8105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:03:40.830366 8105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:03:40.830471 8105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:40.830631 8105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:03:40.830691 8105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 07:03:40 dist-test-slave-69rg krb5kdc[7634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518220, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:03:40.853335 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:40.832124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:43605 (local address 127.6.61.62:44745)
0812 07:03:40.832296 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:40.832303 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:40.832520 (+ 217us) server_negotiation.cc:407] Connection header received
0812 07:03:40.834586 (+ 2066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:40.834592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:40.834682 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:40.834797 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:40.836898 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:40.838051 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:40.840974 (+ 2923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:40.841596 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:40.841752 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:40.845615 (+ 3863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:40.845624 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:40.845631 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:40.845683 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:40.849053 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:40.850221 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:40.850228 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:40.850231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:40.850417 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:40.851044 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:40.851047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:40.851049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:40.851406 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:40.851581 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:40.852812 (+ 1231us) server_negotiation.cc:299] Negotiation successful
0812 07:03:40.853066 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 07:03:40.855294 8133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:40.831435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:44745 (local address 127.6.61.3:43605)
0812 07:03:40.832373 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:40.832418 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:40.834311 (+ 1893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:40.835037 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:40.835048 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:40.835806 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:03:40.836707 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:40.836730 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:40.838229 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:40.838242 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:40.840740 (+ 2498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:40.840759 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:40.841880 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:40.841890 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:40.842121 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:40.842825 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:03:40.842852 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:03:40.845385 (+ 2533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:03:40.849201 (+ 3816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:40.849210 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:40.849227 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:40.850057 (+ 830us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:40.850569 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:40.850578 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:40.850584 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:40.850911 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:40.851578 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:03:40.851589 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:03:40.852238 (+ 649us) client_negotiation.cc:769] Sending connection context
0812 07:03:40.852635 (+ 397us) client_negotiation.cc:240] Negotiation successful
0812 07:03:40.853375 (+ 740us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":367,"threads_started":1}
I20260812 07:03:40.879367 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:40.862774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:39173 (local address 127.6.61.62:44745)
0812 07:03:40.863020 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:40.863028 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:40.864035 (+ 1007us) server_negotiation.cc:407] Connection header received
0812 07:03:40.864295 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:40.864302 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:40.864401 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:40.864525 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:40.866227 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:40.867100 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:40.869791 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:40.870410 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:40.870553 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:40.873020 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:40.873025 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:40.873028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:40.873059 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:40.876476 (+ 3417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:40.877177 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:40.877183 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:40.877187 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:40.877319 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:40.877700 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:40.877705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:40.877708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:40.878076 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:40.878252 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:40.878746 (+ 494us) server_negotiation.cc:299] Negotiation successful
0812 07:03:40.878991 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 07:03:40.879556 8207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:40.862889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:44745 (local address 127.6.61.3:39173)
0812 07:03:40.863903 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:40.863929 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:40.864105 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:40.864581 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:40.864590 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:40.865426 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:03:40.866016 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:40.866035 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:40.867261 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:40.867268 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:40.869599 (+ 2331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:40.869622 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:40.870787 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:40.870792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:40.870919 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:40.871648 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:03:40.871679 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:03:40.872751 (+ 1072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:03:40.876615 (+ 3864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:40.876622 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:40.876626 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:40.877049 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:40.877436 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:40.877441 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:40.877445 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:40.877581 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:40.878229 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:03:40.878236 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:03:40.878448 (+ 212us) client_negotiation.cc:769] Sending connection context
0812 07:03:40.878625 (+ 177us) client_negotiation.cc:240] Negotiation successful
0812 07:03:40.879017 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"thread_start_us":599,"threads_started":1}
I20260812 07:03:40.890933 8105 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.3:35519
I20260812 07:03:40.891077 8260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.3:35519 every 8 connection(s)
I20260812 07:03:40.892544 8105 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
I20260812 07:03:40.894842 8261 heartbeater.cc:347] Connected to a master server at 127.6.61.62:44745
I20260812 07:03:40.895043 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 8105
I20260812 07:03:40.895125 8261 heartbeater.cc:464] Registering TS with master...
I20260812 07:03:40.895259 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance
I20260812 07:03:40.895700 6388 external_mini_cluster.cc:1428] Setting key f532c51a09f5309c58afa3917ac096ee
I20260812 07:03:40.896046 8261 heartbeater.cc:511] Master 127.6.61.62:44745 requested a full tablet report, sending...
I20260812 07:03:40.897608 7683 ts_manager.cc:194] Registered new tserver with Master: fb75cd25c1f045a4a036e627b374310a (127.6.61.3:35519)
I20260812 07:03:40.898713 7683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.3@KRBTEST.COM'} at 127.6.61.3:43605
I20260812 07:03:40.899289 6388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 07:03:40 dist-test-slave-69rg krb5kdc[7634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518220, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 07:03:40 dist-test-slave-69rg krb5kdc[7634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518220, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:03:40.947954 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:40.934380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38136 (local address 127.6.61.62:44745)
0812 07:03:40.934555 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:40.934560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:40.934871 (+ 311us) server_negotiation.cc:407] Connection header received
0812 07:03:40.935024 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:40.935028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:40.935103 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:40.935191 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:40.936726 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:40.937505 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:40.938871 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:40.939288 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:40.939389 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:40.943116 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:40.943121 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:40.943124 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:40.943157 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:40.945608 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:40.946296 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:40.946300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:40.946303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:40.946433 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:40.946783 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:40.946786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:40.946788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:40.947037 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:40.947225 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:40.947478 (+ 253us) server_negotiation.cc:299] Negotiation successful
0812 07:03:40.947652 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 07:03:40.952016 7683 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:03:40.959048 7683 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:03:40.997090 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:40.971873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53766 (local address 127.6.61.1:38735)
0812 07:03:40.985796 (+ 13923us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:40.985831 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:40.985876 (+ 45us) server_negotiation.cc:407] Connection header received
0812 07:03:40.986019 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:40.986032 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:40.986768 (+ 736us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:40.987057 (+ 289us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:03:40.988372 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:40.990558 (+ 2186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:40.994341 (+ 3783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:40.995873 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:40.996008 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:40.996211 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:40.996600 (+ 389us) server_negotiation.cc:299] Negotiation successful
0812 07:03:40.996767 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13653,"thread_start_us":14244,"threads_started":1}
I20260812 07:03:40.997329 8283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:40.971505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:38735 (local address 127.0.0.1:53766)
0812 07:03:40.980251 (+ 8746us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:40.980300 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:40.980518 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:40.987414 (+ 6896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:40.987426 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:40.987474 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:03:40.988064 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:40.988082 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:40.991315 (+ 3233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:40.991322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:40.994123 (+ 2801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:40.994138 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:40.996125 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:40.996140 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:40.996344 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:40.996396 (+ 52us) client_negotiation.cc:769] Sending connection context
0812 07:03:40.996556 (+ 160us) client_negotiation.cc:240] Negotiation successful
0812 07:03:40.996749 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8463,"mutex_wait_us":19,"thread_start_us":18107,"threads_started":1}
I20260812 07:03:40.998684 8276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:40.974544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:36919 (local address 127.0.0.1:33038)
0812 07:03:40.989951 (+ 15407us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:40.989969 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:40.990213 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:40.991190 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:40.991197 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:40.991226 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:03:40.991799 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:40.991818 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:40.993655 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:40.993661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:40.995603 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:40.995617 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:40.997853 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:40.997861 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:40.998047 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:40.998106 (+ 59us) client_negotiation.cc:769] Sending connection context
0812 07:03:40.998219 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 07:03:40.998356 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15238,"thread_start_us":5245,"threads_started":1}
I20260812 07:03:40.998951 8285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:40.975399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33038 (local address 127.6.61.2:36919)
0812 07:03:40.989176 (+ 13777us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:40.989189 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:40.990096 (+ 907us) server_negotiation.cc:407] Connection header received
0812 07:03:40.990443 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:40.990452 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:40.990849 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:40.991020 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:03:40.992055 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:40.993437 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:40.995767 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:40.997156 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:40.997756 (+ 600us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:40.998010 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:40.998283 (+ 273us) server_negotiation.cc:299] Negotiation successful
0812 07:03:40.998454 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13555,"thread_start_us":13787,"threads_started":1}
I20260812 07:03:41.000701 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:40.971864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:35519 (local address 127.0.0.1:46428)
0812 07:03:40.983144 (+ 11280us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:40.983173 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:40.983354 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:40.991970 (+ 8616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:40.991978 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:40.992011 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:03:40.992622 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:40.992635 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:40.994770 (+ 2135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:40.994781 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:40.997293 (+ 2512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:40.997380 (+ 87us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:40.999822 (+ 2442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:40.999829 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:41.000047 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:41.000079 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 07:03:41.000197 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 07:03:41.000345 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11050,"mutex_wait_us":47,"thread_start_us":10834,"threads_started":1}
I20260812 07:03:41.000856 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:40.972112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46428 (local address 127.6.61.3:35519)
0812 07:03:40.991339 (+ 19227us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:40.991373 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:40.991407 (+ 34us) server_negotiation.cc:407] Connection header received
0812 07:03:40.991498 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:40.991503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:40.991707 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:40.992182 (+ 475us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:03:40.992908 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:40.994588 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:40.998138 (+ 3550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:40.999594 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:40.999758 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:40.999996 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:41.000267 (+ 271us) server_negotiation.cc:299] Negotiation successful
0812 07:03:41.000438 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19034,"thread_start_us":18814,"threads_started":1}
I20260812 07:03:41.001986 7852 tablet_service.cc:1467] Processing CreateTablet for tablet 7b086304d1e941d2a8900516f2b52642 (DEFAULT_TABLE table=test-table [id=4943e9a25dca40738c8ad45003bff7d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:03:41.002388 8019 tablet_service.cc:1467] Processing CreateTablet for tablet 7b086304d1e941d2a8900516f2b52642 (DEFAULT_TABLE table=test-table [id=4943e9a25dca40738c8ad45003bff7d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:03:41.003399 7852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b086304d1e941d2a8900516f2b52642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:03:41.003768 8019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b086304d1e941d2a8900516f2b52642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:03:41.004308 8181 tablet_service.cc:1467] Processing CreateTablet for tablet 7b086304d1e941d2a8900516f2b52642 (DEFAULT_TABLE table=test-table [id=4943e9a25dca40738c8ad45003bff7d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:03:41.005710 8181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b086304d1e941d2a8900516f2b52642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:03:41.015501 8286 tablet_bootstrap.cc:492] T 7b086304d1e941d2a8900516f2b52642 P 5e52441d04c64aa4bdcf862cdfbf23c9: Bootstrap starting.
I20260812 07:03:41.019052 8286 tablet_bootstrap.cc:654] T 7b086304d1e941d2a8900516f2b52642 P 5e52441d04c64aa4bdcf862cdfbf23c9: Neither blocks nor log segments found. Creating new log.
I20260812 07:03:41.020599 8287 tablet_bootstrap.cc:492] T 7b086304d1e941d2a8900516f2b52642 P c335f4241d1f4edcb6a58d348f3e7736: Bootstrap starting.
I20260812 07:03:41.020561 8286 log.cc:824] T 7b086304d1e941d2a8900516f2b52642 P 5e52441d04c64aa4bdcf862cdfbf23c9: Log is configured to *not* fsync() on all Append() calls
I20260812 07:03:41.022693 8288 tablet_bootstrap.cc:492] T 7b086304d1e941d2a8900516f2b52642 P fb75cd25c1f045a4a036e627b374310a: Bootstrap starting.
I20260812 07:03:41.024282 8286 tablet_bootstrap.cc:492] T 7b086304d1e941d2a8900516f2b52642 P 5e52441d04c64aa4bdcf862cdfbf23c9: No bootstrap required, opened a new log
I20260812 07:03:41.024583 8286 ts_tablet_manager.cc:1397] T 7b086304d1e941d2a8900516f2b52642 P 5e52441d04c64aa4bdcf862cdfbf23c9: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20260812 07:03:41.024942 8287 tablet_bootstrap.cc:654] T 7b086304d1e941d2a8900516f2b52642 P c335f4241d1f4edcb6a58d348f3e7736: Neither blocks nor log segments found. Creating new log.
I20260812 07:03:41.026384 8288 tablet_bootstrap.cc:654] T 7b086304d1e941d2a8900516f2b52642 P fb75cd25c1f045a4a036e627b374310a: Neither blocks nor log segments found. Creating new log.
I20260812 07:03:41.026360 8287 log.cc:824] T 7b086304d1e941d2a8900516f2b52642 P c335f4241d1f4edcb6a58d348f3e7736: Log is configured to *not* fsync() on all Append() calls
I20260812 07:03:41.027604 8288 log.cc:824] T 7b086304d1e941d2a8900516f2b52642 P fb75cd25c1f045a4a036e627b374310a: Log is configured to *not* fsync() on all Append() calls
I20260812 07:03:41.029651 8287 tablet_bootstrap.cc:492] T 7b086304d1e941d2a8900516f2b52642 P c335f4241d1f4edcb6a58d348f3e7736: No bootstrap required, opened a new log
I20260812 07:03:41.029982 8287 ts_tablet_manager.cc:1397] T 7b086304d1e941d2a8900516f2b52642 P c335f4241d1f4edcb6a58d348f3e7736: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260812 07:03:41.030763 8288 tablet_bootstrap.cc:492] T 7b086304d1e941d2a8900516f2b52642 P fb75cd25c1f045a4a036e627b374310a: No bootstrap required, opened a new log
I20260812 07:03:41.031036 8288 ts_tablet_manager.cc:1397] T 7b086304d1e941d2a8900516f2b52642 P fb75cd25c1f045a4a036e627b374310a: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260812 07:03:41.034466 8286 raft_consensus.cc:348] T 7b086304d1e941d2a8900516f2b52642 P 5e52441d04c64aa4bdcf862cdfbf23c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } }
I20260812 07:03:41.034847 8286 raft_consensus.cc:374] T 7b086304d1e941d2a8900516f2b52642 P 5e52441d04c64aa4bdcf862cdfbf23c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:03:41.034981 8286 raft_consensus.cc:729] T 7b086304d1e941d2a8900516f2b52642 P 5e52441d04c64aa4bdcf862cdfbf23c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e52441d04c64aa4bdcf862cdfbf23c9, State: Initialized, Role: FOLLOWER
I20260812 07:03:41.035617 8286 consensus_queue.cc:260] T 7b086304d1e941d2a8900516f2b52642 P 5e52441d04c64aa4bdcf862cdfbf23c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } }
I20260812 07:03:41.041366 8287 raft_consensus.cc:348] T 7b086304d1e941d2a8900516f2b52642 P c335f4241d1f4edcb6a58d348f3e7736 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } }
I20260812 07:03:41.041833 8287 raft_consensus.cc:374] T 7b086304d1e941d2a8900516f2b52642 P c335f4241d1f4edcb6a58d348f3e7736 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:03:41.041944 8287 raft_consensus.cc:729] T 7b086304d1e941d2a8900516f2b52642 P c335f4241d1f4edcb6a58d348f3e7736 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c335f4241d1f4edcb6a58d348f3e7736, State: Initialized, Role: FOLLOWER
I20260812 07:03:41.042229 8288 raft_consensus.cc:348] T 7b086304d1e941d2a8900516f2b52642 P fb75cd25c1f045a4a036e627b374310a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } }
I20260812 07:03:41.042637 8288 raft_consensus.cc:374] T 7b086304d1e941d2a8900516f2b52642 P fb75cd25c1f045a4a036e627b374310a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:03:41.042749 8288 raft_consensus.cc:729] T 7b086304d1e941d2a8900516f2b52642 P fb75cd25c1f045a4a036e627b374310a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb75cd25c1f045a4a036e627b374310a, State: Initialized, Role: FOLLOWER
I20260812 07:03:41.042678 8287 consensus_queue.cc:260] T 7b086304d1e941d2a8900516f2b52642 P c335f4241d1f4edcb6a58d348f3e7736 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } }
I20260812 07:03:41.043467 8288 consensus_queue.cc:260] T 7b086304d1e941d2a8900516f2b52642 P fb75cd25c1f045a4a036e627b374310a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } }
I20260812 07:03:41.044497 8286 ts_tablet_manager.cc:1428] T 7b086304d1e941d2a8900516f2b52642 P 5e52441d04c64aa4bdcf862cdfbf23c9: Time spent starting tablet: real 0.020s user 0.006s sys 0.015s
I20260812 07:03:41.045708 8101 heartbeater.cc:503] Master 127.6.61.62:44745 was elected leader, sending a full tablet report...
I20260812 07:03:41.045964 8261 heartbeater.cc:503] Master 127.6.61.62:44745 was elected leader, sending a full tablet report...
I20260812 07:03:41.046211 8287 ts_tablet_manager.cc:1428] T 7b086304d1e941d2a8900516f2b52642 P c335f4241d1f4edcb6a58d348f3e7736: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260812 07:03:41.046561 8288 ts_tablet_manager.cc:1428] T 7b086304d1e941d2a8900516f2b52642 P fb75cd25c1f045a4a036e627b374310a: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
W20260812 07:03:41.069907 7937 tablet.cc:2369] T 7b086304d1e941d2a8900516f2b52642 P 5e52441d04c64aa4bdcf862cdfbf23c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 07:03:41.106776 8102 tablet.cc:2369] T 7b086304d1e941d2a8900516f2b52642 P c335f4241d1f4edcb6a58d348f3e7736: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 07:03:41.144232 8262 tablet.cc:2369] T 7b086304d1e941d2a8900516f2b52642 P fb75cd25c1f045a4a036e627b374310a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:03:41.159210 8293 raft_consensus.cc:482] T 7b086304d1e941d2a8900516f2b52642 P c335f4241d1f4edcb6a58d348f3e7736 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:03:41.159543 8293 raft_consensus.cc:504] T 7b086304d1e941d2a8900516f2b52642 P c335f4241d1f4edcb6a58d348f3e7736 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } }
I20260812 07:03:41.161445 8293 leader_election.cc:288] T 7b086304d1e941d2a8900516f2b52642 P c335f4241d1f4edcb6a58d348f3e7736 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e52441d04c64aa4bdcf862cdfbf23c9 (127.6.61.1:38735), fb75cd25c1f045a4a036e627b374310a (127.6.61.3:35519)
I20260812 07:03:41.170670 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:41.161909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:53355 (local address 127.6.61.3:35519)
0812 07:03:41.162408 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:41.162417 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:41.163182 (+ 765us) server_negotiation.cc:407] Connection header received
0812 07:03:41.163434 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:41.163440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:41.163514 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:41.163587 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:03:41.164341 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.165376 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.168347 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.169593 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.169727 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:41.169872 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:41.170248 (+ 376us) server_negotiation.cc:299] Negotiation successful
0812 07:03:41.170399 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297}
I20260812 07:03:41.170796 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:41.161787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:35519 (local address 127.6.61.2:53355)
0812 07:03:41.163064 (+ 1277us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:41.163083 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:41.163249 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:41.163660 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:41.163664 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:41.163690 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:03:41.164169 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:41.164180 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.165524 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.165530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:41.168144 (+ 2614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:41.168167 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.169898 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.169905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:41.170040 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:41.170091 (+ 51us) client_negotiation.cc:769] Sending connection context
0812 07:03:41.170189 (+ 98us) client_negotiation.cc:240] Negotiation successful
0812 07:03:41.170297 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1111,"mutex_wait_us":95,"thread_start_us":394,"threads_started":1}
I20260812 07:03:41.170912 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:41.161862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:51893 (local address 127.6.61.1:38735)
0812 07:03:41.162164 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:41.162174 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:41.162784 (+ 610us) server_negotiation.cc:407] Connection header received
0812 07:03:41.163107 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:41.163111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:41.163189 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:41.163299 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:03:41.164097 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.165210 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.168347 (+ 3137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.169605 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.169727 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:41.169865 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:41.170405 (+ 540us) server_negotiation.cc:299] Negotiation successful
0812 07:03:41.170561 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 07:03:41.171471 8296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:41.161620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:38735 (local address 127.6.61.2:51893)
0812 07:03:41.162645 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:41.162672 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:41.162932 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:41.163346 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:41.163351 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:41.163383 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:03:41.163920 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:41.163937 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.165370 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.165375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:41.168167 (+ 2792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:41.168177 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.170020 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.170025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:41.170217 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:41.170259 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 07:03:41.170350 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 07:03:41.170475 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":838,"mutex_wait_us":119,"thread_start_us":1053,"threads_started":1}
I20260812 07:03:41.171903 8214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b086304d1e941d2a8900516f2b52642" candidate_uuid: "c335f4241d1f4edcb6a58d348f3e7736" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb75cd25c1f045a4a036e627b374310a" is_pre_election: true
I20260812 07:03:41.172562 8214 raft_consensus.cc:2454] T 7b086304d1e941d2a8900516f2b52642 P fb75cd25c1f045a4a036e627b374310a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c335f4241d1f4edcb6a58d348f3e7736 in term 0.
I20260812 07:03:41.172554 7881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b086304d1e941d2a8900516f2b52642" candidate_uuid: "c335f4241d1f4edcb6a58d348f3e7736" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" is_pre_election: true
I20260812 07:03:41.173110 7881 raft_consensus.cc:2454] T 7b086304d1e941d2a8900516f2b52642 P 5e52441d04c64aa4bdcf862cdfbf23c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c335f4241d1f4edcb6a58d348f3e7736 in term 0.
I20260812 07:03:41.173286 7962 leader_election.cc:302] T 7b086304d1e941d2a8900516f2b52642 P c335f4241d1f4edcb6a58d348f3e7736 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c335f4241d1f4edcb6a58d348f3e7736, fb75cd25c1f045a4a036e627b374310a; no voters:
I20260812 07:03:41.173712 8293 raft_consensus.cc:2781] T 7b086304d1e941d2a8900516f2b52642 P c335f4241d1f4edcb6a58d348f3e7736 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:03:41.173942 8293 raft_consensus.cc:482] T 7b086304d1e941d2a8900516f2b52642 P c335f4241d1f4edcb6a58d348f3e7736 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:03:41.174065 8293 raft_consensus.cc:3037] T 7b086304d1e941d2a8900516f2b52642 P c335f4241d1f4edcb6a58d348f3e7736 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:03:41.176563 8293 raft_consensus.cc:504] T 7b086304d1e941d2a8900516f2b52642 P c335f4241d1f4edcb6a58d348f3e7736 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } }
I20260812 07:03:41.177202 8293 leader_election.cc:288] T 7b086304d1e941d2a8900516f2b52642 P c335f4241d1f4edcb6a58d348f3e7736 [CANDIDATE]: Term 1 election: Requested vote from peers 5e52441d04c64aa4bdcf862cdfbf23c9 (127.6.61.1:38735), fb75cd25c1f045a4a036e627b374310a (127.6.61.3:35519)
I20260812 07:03:41.177663 7881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b086304d1e941d2a8900516f2b52642" candidate_uuid: "c335f4241d1f4edcb6a58d348f3e7736" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9"
I20260812 07:03:41.177673 8214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b086304d1e941d2a8900516f2b52642" candidate_uuid: "c335f4241d1f4edcb6a58d348f3e7736" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb75cd25c1f045a4a036e627b374310a"
I20260812 07:03:41.177904 8214 raft_consensus.cc:3037] T 7b086304d1e941d2a8900516f2b52642 P fb75cd25c1f045a4a036e627b374310a [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:03:41.177904 7881 raft_consensus.cc:3037] T 7b086304d1e941d2a8900516f2b52642 P 5e52441d04c64aa4bdcf862cdfbf23c9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:03:41.181319 7881 raft_consensus.cc:2454] T 7b086304d1e941d2a8900516f2b52642 P 5e52441d04c64aa4bdcf862cdfbf23c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c335f4241d1f4edcb6a58d348f3e7736 in term 1.
I20260812 07:03:41.181320 8214 raft_consensus.cc:2454] T 7b086304d1e941d2a8900516f2b52642 P fb75cd25c1f045a4a036e627b374310a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c335f4241d1f4edcb6a58d348f3e7736 in term 1.
I20260812 07:03:41.181809 7960 leader_election.cc:302] T 7b086304d1e941d2a8900516f2b52642 P c335f4241d1f4edcb6a58d348f3e7736 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e52441d04c64aa4bdcf862cdfbf23c9, c335f4241d1f4edcb6a58d348f3e7736; no voters:
I20260812 07:03:41.182298 8293 raft_consensus.cc:2781] T 7b086304d1e941d2a8900516f2b52642 P c335f4241d1f4edcb6a58d348f3e7736 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:03:41.182758 8293 raft_consensus.cc:686] T 7b086304d1e941d2a8900516f2b52642 P c335f4241d1f4edcb6a58d348f3e7736 [term 1 LEADER]: Becoming Leader. State: Replica: c335f4241d1f4edcb6a58d348f3e7736, State: Running, Role: LEADER
I20260812 07:03:41.183279 8293 consensus_queue.cc:237] T 7b086304d1e941d2a8900516f2b52642 P c335f4241d1f4edcb6a58d348f3e7736 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } }
I20260812 07:03:41.188583 7683 catalog_manager.cc:5495] T 7b086304d1e941d2a8900516f2b52642 P c335f4241d1f4edcb6a58d348f3e7736 reported cstate change: term changed from 0 to 1, leader changed from <none> to c335f4241d1f4edcb6a58d348f3e7736 (127.6.61.2). New cstate: current_term: 1 leader_uuid: "c335f4241d1f4edcb6a58d348f3e7736" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } health_report { overall_health: HEALTHY } } }
I20260812 07:03:41.213019 8285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:41.207224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33048 (local address 127.6.61.2:36919)
0812 07:03:41.207433 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:41.207437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:41.207456 (+ 19us) server_negotiation.cc:407] Connection header received
0812 07:03:41.207696 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:41.207700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:41.207784 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:41.207904 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:03:41.208281 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.209023 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.210488 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.210966 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.211082 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:41.211676 (+ 594us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:03:41.212265 (+ 589us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:03:41.212392 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:41.212576 (+ 184us) server_negotiation.cc:299] Negotiation successful
0812 07:03:41.212661 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 07:03:41.222519 8214 raft_consensus.cc:1261] T 7b086304d1e941d2a8900516f2b52642 P fb75cd25c1f045a4a036e627b374310a [term 1 FOLLOWER]: Refusing update from remote peer c335f4241d1f4edcb6a58d348f3e7736: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:03:41.222520 7881 raft_consensus.cc:1261] T 7b086304d1e941d2a8900516f2b52642 P 5e52441d04c64aa4bdcf862cdfbf23c9 [term 1 FOLLOWER]: Refusing update from remote peer c335f4241d1f4edcb6a58d348f3e7736: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:03:41.223592 8298 consensus_queue.cc:1035] T 7b086304d1e941d2a8900516f2b52642 P c335f4241d1f4edcb6a58d348f3e7736 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:03:41.224016 8293 consensus_queue.cc:1035] T 7b086304d1e941d2a8900516f2b52642 P c335f4241d1f4edcb6a58d348f3e7736 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:03:41.236802 8302 mvcc.cc:205] Tried to move back new op lower bound from 7317578634116288512 to 7317578633971048448. Current Snapshot: MvccSnapshot[applied={T|T < 7317578634116288512}]
I20260812 07:03:41.256803 7682 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:03:41.257227 7682 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:03:41.261582 7682 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f1dea41f34141ffa6efc53e55582c43: Sending DeleteTablet for 3 replicas of tablet 7b086304d1e941d2a8900516f2b52642
I20260812 07:03:41.262948 8181 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b086304d1e941d2a8900516f2b52642 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:03:41 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:46428
I20260812 07:03:41.262997 7852 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b086304d1e941d2a8900516f2b52642 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:03:41 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:53766
I20260812 07:03:41.263315 8019 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b086304d1e941d2a8900516f2b52642 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:03:41 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:33038
I20260812 07:03:41.265686 7682 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:03:41.267410 7682 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:03:41.273885 8315 tablet_replica.cc:330] T 7b086304d1e941d2a8900516f2b52642 P 5e52441d04c64aa4bdcf862cdfbf23c9: stopping tablet replica
I20260812 07:03:41.274386 8315 raft_consensus.cc:2229] T 7b086304d1e941d2a8900516f2b52642 P 5e52441d04c64aa4bdcf862cdfbf23c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:03:41.274952 8315 raft_consensus.cc:2258] T 7b086304d1e941d2a8900516f2b52642 P 5e52441d04c64aa4bdcf862cdfbf23c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:03:41.278465 8180 tablet_service.cc:1467] Processing CreateTablet for tablet 5a5f3b50042c4bcbb7a93149b83c21b7 (DEFAULT_TABLE table=test-table [id=1e2eff09f95846158facbde0a926ff30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:03:41.278504 8018 tablet_service.cc:1467] Processing CreateTablet for tablet 5a5f3b50042c4bcbb7a93149b83c21b7 (DEFAULT_TABLE table=test-table [id=1e2eff09f95846158facbde0a926ff30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:03:41.279120 8180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a5f3b50042c4bcbb7a93149b83c21b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:03:41.279140 8018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a5f3b50042c4bcbb7a93149b83c21b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:03:41.279235 7852 tablet_service.cc:1467] Processing CreateTablet for tablet 5a5f3b50042c4bcbb7a93149b83c21b7 (DEFAULT_TABLE table=test-table [id=1e2eff09f95846158facbde0a926ff30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:03:41.279673 7852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a5f3b50042c4bcbb7a93149b83c21b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:03:41.286410 8286 tablet_bootstrap.cc:492] T 5a5f3b50042c4bcbb7a93149b83c21b7 P 5e52441d04c64aa4bdcf862cdfbf23c9: Bootstrap starting.
I20260812 07:03:41.286438 8287 tablet_bootstrap.cc:492] T 5a5f3b50042c4bcbb7a93149b83c21b7 P c335f4241d1f4edcb6a58d348f3e7736: Bootstrap starting.
I20260812 07:03:41.286459 8288 tablet_bootstrap.cc:492] T 5a5f3b50042c4bcbb7a93149b83c21b7 P fb75cd25c1f045a4a036e627b374310a: Bootstrap starting.
I20260812 07:03:41.288208 8287 tablet_bootstrap.cc:654] T 5a5f3b50042c4bcbb7a93149b83c21b7 P c335f4241d1f4edcb6a58d348f3e7736: Neither blocks nor log segments found. Creating new log.
I20260812 07:03:41.288208 8286 tablet_bootstrap.cc:654] T 5a5f3b50042c4bcbb7a93149b83c21b7 P 5e52441d04c64aa4bdcf862cdfbf23c9: Neither blocks nor log segments found. Creating new log.
I20260812 07:03:41.288208 8288 tablet_bootstrap.cc:654] T 5a5f3b50042c4bcbb7a93149b83c21b7 P fb75cd25c1f045a4a036e627b374310a: Neither blocks nor log segments found. Creating new log.
I20260812 07:03:41.300374 8288 tablet_bootstrap.cc:492] T 5a5f3b50042c4bcbb7a93149b83c21b7 P fb75cd25c1f045a4a036e627b374310a: No bootstrap required, opened a new log
I20260812 07:03:41.300607 8288 ts_tablet_manager.cc:1397] T 5a5f3b50042c4bcbb7a93149b83c21b7 P fb75cd25c1f045a4a036e627b374310a: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20260812 07:03:41.301532 8288 raft_consensus.cc:348] T 5a5f3b50042c4bcbb7a93149b83c21b7 P fb75cd25c1f045a4a036e627b374310a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } }
I20260812 07:03:41.301680 8315 ts_tablet_manager.cc:1905] T 7b086304d1e941d2a8900516f2b52642 P 5e52441d04c64aa4bdcf862cdfbf23c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:03:41.301930 8288 raft_consensus.cc:374] T 5a5f3b50042c4bcbb7a93149b83c21b7 P fb75cd25c1f045a4a036e627b374310a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:03:41.302096 8288 raft_consensus.cc:729] T 5a5f3b50042c4bcbb7a93149b83c21b7 P fb75cd25c1f045a4a036e627b374310a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb75cd25c1f045a4a036e627b374310a, State: Initialized, Role: FOLLOWER
I20260812 07:03:41.302460 8288 consensus_queue.cc:260] T 5a5f3b50042c4bcbb7a93149b83c21b7 P fb75cd25c1f045a4a036e627b374310a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } }
I20260812 07:03:41.303213 8288 ts_tablet_manager.cc:1428] T 5a5f3b50042c4bcbb7a93149b83c21b7 P fb75cd25c1f045a4a036e627b374310a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260812 07:03:41.305251 8317 tablet_replica.cc:330] T 7b086304d1e941d2a8900516f2b52642 P c335f4241d1f4edcb6a58d348f3e7736: stopping tablet replica
I20260812 07:03:41.305861 8317 raft_consensus.cc:2229] T 7b086304d1e941d2a8900516f2b52642 P c335f4241d1f4edcb6a58d348f3e7736 [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:03:41.306720 8317 raft_consensus.cc:2258] T 7b086304d1e941d2a8900516f2b52642 P c335f4241d1f4edcb6a58d348f3e7736 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:03:41.306905 8287 tablet_bootstrap.cc:492] T 5a5f3b50042c4bcbb7a93149b83c21b7 P c335f4241d1f4edcb6a58d348f3e7736: No bootstrap required, opened a new log
I20260812 07:03:41.307375 8287 ts_tablet_manager.cc:1397] T 5a5f3b50042c4bcbb7a93149b83c21b7 P c335f4241d1f4edcb6a58d348f3e7736: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.004s
I20260812 07:03:41.308336 8287 raft_consensus.cc:348] T 5a5f3b50042c4bcbb7a93149b83c21b7 P c335f4241d1f4edcb6a58d348f3e7736 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } }
I20260812 07:03:41.308635 8287 raft_consensus.cc:374] T 5a5f3b50042c4bcbb7a93149b83c21b7 P c335f4241d1f4edcb6a58d348f3e7736 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:03:41.308792 8287 raft_consensus.cc:729] T 5a5f3b50042c4bcbb7a93149b83c21b7 P c335f4241d1f4edcb6a58d348f3e7736 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c335f4241d1f4edcb6a58d348f3e7736, State: Initialized, Role: FOLLOWER
I20260812 07:03:41.309128 8287 consensus_queue.cc:260] T 5a5f3b50042c4bcbb7a93149b83c21b7 P c335f4241d1f4edcb6a58d348f3e7736 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } }
I20260812 07:03:41.309474 8317 ts_tablet_manager.cc:1905] T 7b086304d1e941d2a8900516f2b52642 P c335f4241d1f4edcb6a58d348f3e7736: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:03:41.309870 8287 ts_tablet_manager.cc:1428] T 5a5f3b50042c4bcbb7a93149b83c21b7 P c335f4241d1f4edcb6a58d348f3e7736: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260812 07:03:41.313954 8316 tablet_replica.cc:330] T 7b086304d1e941d2a8900516f2b52642 P fb75cd25c1f045a4a036e627b374310a: stopping tablet replica
I20260812 07:03:41.314541 8316 raft_consensus.cc:2229] T 7b086304d1e941d2a8900516f2b52642 P fb75cd25c1f045a4a036e627b374310a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:03:41.315165 8316 raft_consensus.cc:2258] T 7b086304d1e941d2a8900516f2b52642 P fb75cd25c1f045a4a036e627b374310a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:03:41.317008 8286 tablet_bootstrap.cc:492] T 5a5f3b50042c4bcbb7a93149b83c21b7 P 5e52441d04c64aa4bdcf862cdfbf23c9: No bootstrap required, opened a new log
I20260812 07:03:41.317281 8286 ts_tablet_manager.cc:1397] T 5a5f3b50042c4bcbb7a93149b83c21b7 P 5e52441d04c64aa4bdcf862cdfbf23c9: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.012s
I20260812 07:03:41.318117 8316 ts_tablet_manager.cc:1905] T 7b086304d1e941d2a8900516f2b52642 P fb75cd25c1f045a4a036e627b374310a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:03:41.318367 8286 raft_consensus.cc:348] T 5a5f3b50042c4bcbb7a93149b83c21b7 P 5e52441d04c64aa4bdcf862cdfbf23c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } }
I20260812 07:03:41.318674 8286 raft_consensus.cc:374] T 5a5f3b50042c4bcbb7a93149b83c21b7 P 5e52441d04c64aa4bdcf862cdfbf23c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:03:41.318823 8286 raft_consensus.cc:729] T 5a5f3b50042c4bcbb7a93149b83c21b7 P 5e52441d04c64aa4bdcf862cdfbf23c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e52441d04c64aa4bdcf862cdfbf23c9, State: Initialized, Role: FOLLOWER
I20260812 07:03:41.319240 8286 consensus_queue.cc:260] T 5a5f3b50042c4bcbb7a93149b83c21b7 P 5e52441d04c64aa4bdcf862cdfbf23c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } }
I20260812 07:03:41.319991 8286 ts_tablet_manager.cc:1428] T 5a5f3b50042c4bcbb7a93149b83c21b7 P 5e52441d04c64aa4bdcf862cdfbf23c9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 07:03:41.322770 8315 ts_tablet_manager.cc:1918] T 7b086304d1e941d2a8900516f2b52642 P 5e52441d04c64aa4bdcf862cdfbf23c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:03:41.322963 8315 log.cc:1196] T 7b086304d1e941d2a8900516f2b52642 P 5e52441d04c64aa4bdcf862cdfbf23c9: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-0/wal/wals/7b086304d1e941d2a8900516f2b52642
I20260812 07:03:41.323534 8315 ts_tablet_manager.cc:1939] T 7b086304d1e941d2a8900516f2b52642 P 5e52441d04c64aa4bdcf862cdfbf23c9: Deleting consensus metadata
I20260812 07:03:41.325491 7666 catalog_manager.cc:4841] TS 5e52441d04c64aa4bdcf862cdfbf23c9 (127.6.61.1:38735): tablet 7b086304d1e941d2a8900516f2b52642 (table test-table [id=4943e9a25dca40738c8ad45003bff7d9]) successfully deleted
I20260812 07:03:41.325552 8316 ts_tablet_manager.cc:1918] T 7b086304d1e941d2a8900516f2b52642 P fb75cd25c1f045a4a036e627b374310a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:03:41.325723 8316 log.cc:1196] T 7b086304d1e941d2a8900516f2b52642 P fb75cd25c1f045a4a036e627b374310a: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-2/wal/wals/7b086304d1e941d2a8900516f2b52642
I20260812 07:03:41.326225 8316 ts_tablet_manager.cc:1939] T 7b086304d1e941d2a8900516f2b52642 P fb75cd25c1f045a4a036e627b374310a: Deleting consensus metadata
I20260812 07:03:41.326996 8317 ts_tablet_manager.cc:1918] T 7b086304d1e941d2a8900516f2b52642 P c335f4241d1f4edcb6a58d348f3e7736: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:03:41.327193 8317 log.cc:1196] T 7b086304d1e941d2a8900516f2b52642 P c335f4241d1f4edcb6a58d348f3e7736: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-1/wal/wals/7b086304d1e941d2a8900516f2b52642
I20260812 07:03:41.327777 8317 ts_tablet_manager.cc:1939] T 7b086304d1e941d2a8900516f2b52642 P c335f4241d1f4edcb6a58d348f3e7736: Deleting consensus metadata
I20260812 07:03:41.328027 7668 catalog_manager.cc:4841] TS fb75cd25c1f045a4a036e627b374310a (127.6.61.3:35519): tablet 7b086304d1e941d2a8900516f2b52642 (table test-table [id=4943e9a25dca40738c8ad45003bff7d9]) successfully deleted
I20260812 07:03:41.329334 7667 catalog_manager.cc:4841] TS c335f4241d1f4edcb6a58d348f3e7736 (127.6.61.2:36919): tablet 7b086304d1e941d2a8900516f2b52642 (table test-table [id=4943e9a25dca40738c8ad45003bff7d9]) successfully deleted
I20260812 07:03:41.446257 8292 raft_consensus.cc:482] T 5a5f3b50042c4bcbb7a93149b83c21b7 P 5e52441d04c64aa4bdcf862cdfbf23c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:03:41.446529 8292 raft_consensus.cc:504] T 5a5f3b50042c4bcbb7a93149b83c21b7 P 5e52441d04c64aa4bdcf862cdfbf23c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } }
I20260812 07:03:41.448236 8292 leader_election.cc:288] T 5a5f3b50042c4bcbb7a93149b83c21b7 P 5e52441d04c64aa4bdcf862cdfbf23c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c335f4241d1f4edcb6a58d348f3e7736 (127.6.61.2:36919), fb75cd25c1f045a4a036e627b374310a (127.6.61.3:35519)
I20260812 07:03:41.456082 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:41.448998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:39265 (local address 127.6.61.3:35519)
0812 07:03:41.449181 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:41.449197 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:41.449981 (+ 784us) server_negotiation.cc:407] Connection header received
0812 07:03:41.450281 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:41.450287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:41.450374 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:41.450439 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:03:41.451258 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.452237 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.454107 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.455060 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.455205 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:41.455304 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:41.455676 (+ 372us) server_negotiation.cc:299] Negotiation successful
0812 07:03:41.455813 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 07:03:41.456158 8323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:41.448613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:35519 (local address 127.6.61.1:39265)
0812 07:03:41.449872 (+ 1259us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:41.449890 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:41.450097 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:41.450646 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:41.450653 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:41.450680 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:03:41.451075 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:41.451087 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.452377 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.452389 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:41.453951 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:41.453964 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.455295 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.455302 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:41.455493 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:41.455530 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 07:03:41.455629 (+ 99us) client_negotiation.cc:240] Negotiation successful
0812 07:03:41.455767 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1105,"thread_start_us":574,"threads_started":1}
I20260812 07:03:41.457149 8285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:41.448626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:53013 (local address 127.6.61.2:36919)
0812 07:03:41.448960 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:41.448970 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:41.449656 (+ 686us) server_negotiation.cc:407] Connection header received
0812 07:03:41.450023 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:41.450027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:41.450091 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:41.450190 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:03:41.450761 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.451891 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.454609 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.455825 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.455969 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:41.456055 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:41.456664 (+ 609us) server_negotiation.cc:299] Negotiation successful
0812 07:03:41.456800 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260812 07:03:41.457036 8214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a5f3b50042c4bcbb7a93149b83c21b7" candidate_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb75cd25c1f045a4a036e627b374310a" is_pre_election: true
I20260812 07:03:41.457314 8214 raft_consensus.cc:2454] T 5a5f3b50042c4bcbb7a93149b83c21b7 P fb75cd25c1f045a4a036e627b374310a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e52441d04c64aa4bdcf862cdfbf23c9 in term 0.
I20260812 07:03:41.457753 8322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:41.448592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:36919 (local address 127.6.61.1:53013)
0812 07:03:41.449527 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:41.449560 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:41.449787 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:41.450231 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:41.450236 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:41.450266 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:03:41.450582 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:41.450591 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.452048 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.452054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:41.454440 (+ 2386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:41.454456 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.456253 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.456262 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:41.456421 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:41.456453 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 07:03:41.456553 (+ 100us) client_negotiation.cc:240] Negotiation successful
0812 07:03:41.456699 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":56,"thread_start_us":948,"threads_started":1}
I20260812 07:03:41.457911 7798 leader_election.cc:302] T 5a5f3b50042c4bcbb7a93149b83c21b7 P 5e52441d04c64aa4bdcf862cdfbf23c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e52441d04c64aa4bdcf862cdfbf23c9, fb75cd25c1f045a4a036e627b374310a; no voters:
I20260812 07:03:41.458388 8292 raft_consensus.cc:2781] T 5a5f3b50042c4bcbb7a93149b83c21b7 P 5e52441d04c64aa4bdcf862cdfbf23c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:03:41.458560 8292 raft_consensus.cc:482] T 5a5f3b50042c4bcbb7a93149b83c21b7 P 5e52441d04c64aa4bdcf862cdfbf23c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:03:41.458658 8292 raft_consensus.cc:3037] T 5a5f3b50042c4bcbb7a93149b83c21b7 P 5e52441d04c64aa4bdcf862cdfbf23c9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:03:41.458869 8046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a5f3b50042c4bcbb7a93149b83c21b7" candidate_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c335f4241d1f4edcb6a58d348f3e7736" is_pre_election: true
I20260812 07:03:41.459324 8046 raft_consensus.cc:2454] T 5a5f3b50042c4bcbb7a93149b83c21b7 P c335f4241d1f4edcb6a58d348f3e7736 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e52441d04c64aa4bdcf862cdfbf23c9 in term 0.
I20260812 07:03:41.461337 8292 raft_consensus.cc:504] T 5a5f3b50042c4bcbb7a93149b83c21b7 P 5e52441d04c64aa4bdcf862cdfbf23c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } }
I20260812 07:03:41.461946 8292 leader_election.cc:288] T 5a5f3b50042c4bcbb7a93149b83c21b7 P 5e52441d04c64aa4bdcf862cdfbf23c9 [CANDIDATE]: Term 1 election: Requested vote from peers c335f4241d1f4edcb6a58d348f3e7736 (127.6.61.2:36919), fb75cd25c1f045a4a036e627b374310a (127.6.61.3:35519)
I20260812 07:03:41.462323 8046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a5f3b50042c4bcbb7a93149b83c21b7" candidate_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c335f4241d1f4edcb6a58d348f3e7736"
I20260812 07:03:41.462543 8046 raft_consensus.cc:3037] T 5a5f3b50042c4bcbb7a93149b83c21b7 P c335f4241d1f4edcb6a58d348f3e7736 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:03:41.462699 8214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a5f3b50042c4bcbb7a93149b83c21b7" candidate_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb75cd25c1f045a4a036e627b374310a"
I20260812 07:03:41.462915 8214 raft_consensus.cc:3037] T 5a5f3b50042c4bcbb7a93149b83c21b7 P fb75cd25c1f045a4a036e627b374310a [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:03:41.465930 8214 raft_consensus.cc:2454] T 5a5f3b50042c4bcbb7a93149b83c21b7 P fb75cd25c1f045a4a036e627b374310a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e52441d04c64aa4bdcf862cdfbf23c9 in term 1.
I20260812 07:03:41.465952 8046 raft_consensus.cc:2454] T 5a5f3b50042c4bcbb7a93149b83c21b7 P c335f4241d1f4edcb6a58d348f3e7736 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e52441d04c64aa4bdcf862cdfbf23c9 in term 1.
I20260812 07:03:41.466336 7798 leader_election.cc:302] T 5a5f3b50042c4bcbb7a93149b83c21b7 P 5e52441d04c64aa4bdcf862cdfbf23c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e52441d04c64aa4bdcf862cdfbf23c9, fb75cd25c1f045a4a036e627b374310a; no voters:
I20260812 07:03:41.466666 8292 raft_consensus.cc:2781] T 5a5f3b50042c4bcbb7a93149b83c21b7 P 5e52441d04c64aa4bdcf862cdfbf23c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:03:41.467274 8292 raft_consensus.cc:686] T 5a5f3b50042c4bcbb7a93149b83c21b7 P 5e52441d04c64aa4bdcf862cdfbf23c9 [term 1 LEADER]: Becoming Leader. State: Replica: 5e52441d04c64aa4bdcf862cdfbf23c9, State: Running, Role: LEADER
I20260812 07:03:41.467713 8292 consensus_queue.cc:237] T 5a5f3b50042c4bcbb7a93149b83c21b7 P 5e52441d04c64aa4bdcf862cdfbf23c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } }
I20260812 07:03:41.471724 7682 catalog_manager.cc:5495] T 5a5f3b50042c4bcbb7a93149b83c21b7 P 5e52441d04c64aa4bdcf862cdfbf23c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e52441d04c64aa4bdcf862cdfbf23c9 (127.6.61.1). New cstate: current_term: 1 leader_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } health_report { overall_health: UNKNOWN } } }
Aug 12 07:03:41 dist-test-slave-69rg krb5kdc[7634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518219, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:03:41.561041 8333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:41.544309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:44745 (local address 127.0.0.1:38150)
0812 07:03:41.545079 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:41.545108 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:41.545330 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:41.545766 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:41.545774 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:41.546743 (+ 969us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:03:41.547012 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:41.547026 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.548156 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.548163 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:41.549740 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:41.549758 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.550521 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.550529 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:41.550719 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:41.551470 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:03:41.551496 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:03:41.554274 (+ 2778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:03:41.558268 (+ 3994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:41.558279 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:41.558285 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:41.558870 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:41.559330 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:41.559333 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:41.559335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:41.559485 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:41.560170 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:03:41.560177 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:03:41.560382 (+ 205us) client_negotiation.cc:769] Sending connection context
0812 07:03:41.560517 (+ 135us) client_negotiation.cc:240] Negotiation successful
0812 07:03:41.560731 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":406,"threads_started":1}
I20260812 07:03:41.561412 8334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:41.544388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38150 (local address 127.6.61.62:44745)
0812 07:03:41.545443 (+ 1055us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:41.545448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:41.545467 (+ 19us) server_negotiation.cc:407] Connection header received
0812 07:03:41.545515 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:41.545519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:41.545572 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:41.545661 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:41.547202 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.547994 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.549924 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.550339 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.550436 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:41.554486 (+ 4050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:41.554493 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:41.554498 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:41.554541 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:41.558123 (+ 3582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:41.559023 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:41.559032 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:41.559035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:41.559207 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:41.559622 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:41.559630 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:41.559633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:41.560021 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:41.560179 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:41.560681 (+ 502us) server_negotiation.cc:299] Negotiation successful
0812 07:03:41.560924 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":952,"thread_start_us":820,"threads_started":1}
I20260812 07:03:41.567293 7682 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:38150:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 07:03:41.568638 7682 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:03:41.576735 7852 tablet_service.cc:1467] Processing CreateTablet for tablet e7b449f219fa4590b5a97dc7a7376292 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3173461cb8b24f4aba39dcd7b2261f61]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 07:03:41.576740 8181 tablet_service.cc:1467] Processing CreateTablet for tablet e7b449f219fa4590b5a97dc7a7376292 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3173461cb8b24f4aba39dcd7b2261f61]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 07:03:41.577351 8181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7b449f219fa4590b5a97dc7a7376292. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:03:41.577351 7852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7b449f219fa4590b5a97dc7a7376292. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:03:41.581144 8019 tablet_service.cc:1467] Processing CreateTablet for tablet e7b449f219fa4590b5a97dc7a7376292 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3173461cb8b24f4aba39dcd7b2261f61]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 07:03:41.581779 8019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7b449f219fa4590b5a97dc7a7376292. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:03:41.585798 8288 tablet_bootstrap.cc:492] T e7b449f219fa4590b5a97dc7a7376292 P fb75cd25c1f045a4a036e627b374310a: Bootstrap starting.
I20260812 07:03:41.587280 8286 tablet_bootstrap.cc:492] T e7b449f219fa4590b5a97dc7a7376292 P 5e52441d04c64aa4bdcf862cdfbf23c9: Bootstrap starting.
I20260812 07:03:41.587532 8288 tablet_bootstrap.cc:654] T e7b449f219fa4590b5a97dc7a7376292 P fb75cd25c1f045a4a036e627b374310a: Neither blocks nor log segments found. Creating new log.
I20260812 07:03:41.589056 8286 tablet_bootstrap.cc:654] T e7b449f219fa4590b5a97dc7a7376292 P 5e52441d04c64aa4bdcf862cdfbf23c9: Neither blocks nor log segments found. Creating new log.
I20260812 07:03:41.589576 8288 tablet_bootstrap.cc:492] T e7b449f219fa4590b5a97dc7a7376292 P fb75cd25c1f045a4a036e627b374310a: No bootstrap required, opened a new log
I20260812 07:03:41.589771 8288 ts_tablet_manager.cc:1397] T e7b449f219fa4590b5a97dc7a7376292 P fb75cd25c1f045a4a036e627b374310a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260812 07:03:41.590633 8288 raft_consensus.cc:348] T e7b449f219fa4590b5a97dc7a7376292 P fb75cd25c1f045a4a036e627b374310a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } }
I20260812 07:03:41.590893 8288 raft_consensus.cc:374] T e7b449f219fa4590b5a97dc7a7376292 P fb75cd25c1f045a4a036e627b374310a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:03:41.591048 8288 raft_consensus.cc:729] T e7b449f219fa4590b5a97dc7a7376292 P fb75cd25c1f045a4a036e627b374310a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb75cd25c1f045a4a036e627b374310a, State: Initialized, Role: FOLLOWER
I20260812 07:03:41.591313 8288 consensus_queue.cc:260] T e7b449f219fa4590b5a97dc7a7376292 P fb75cd25c1f045a4a036e627b374310a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } }
I20260812 07:03:41.591912 8288 ts_tablet_manager.cc:1428] T e7b449f219fa4590b5a97dc7a7376292 P fb75cd25c1f045a4a036e627b374310a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 07:03:41.591845 8294 tablet_replica.cc:439] T e7b449f219fa4590b5a97dc7a7376292 P fb75cd25c1f045a4a036e627b374310a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } } }
I20260812 07:03:41.592221 8294 tablet_replica.cc:442] T e7b449f219fa4590b5a97dc7a7376292 P fb75cd25c1f045a4a036e627b374310a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:03:41.594012 8287 tablet_bootstrap.cc:492] T e7b449f219fa4590b5a97dc7a7376292 P c335f4241d1f4edcb6a58d348f3e7736: Bootstrap starting.
I20260812 07:03:41.595374 8286 tablet_bootstrap.cc:492] T e7b449f219fa4590b5a97dc7a7376292 P 5e52441d04c64aa4bdcf862cdfbf23c9: No bootstrap required, opened a new log
I20260812 07:03:41.595562 8286 ts_tablet_manager.cc:1397] T e7b449f219fa4590b5a97dc7a7376292 P 5e52441d04c64aa4bdcf862cdfbf23c9: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260812 07:03:41.595928 8287 tablet_bootstrap.cc:654] T e7b449f219fa4590b5a97dc7a7376292 P c335f4241d1f4edcb6a58d348f3e7736: Neither blocks nor log segments found. Creating new log.
I20260812 07:03:41.596451 8286 raft_consensus.cc:348] T e7b449f219fa4590b5a97dc7a7376292 P 5e52441d04c64aa4bdcf862cdfbf23c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } }
I20260812 07:03:41.596719 8286 raft_consensus.cc:374] T e7b449f219fa4590b5a97dc7a7376292 P 5e52441d04c64aa4bdcf862cdfbf23c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:03:41.596865 8286 raft_consensus.cc:729] T e7b449f219fa4590b5a97dc7a7376292 P 5e52441d04c64aa4bdcf862cdfbf23c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e52441d04c64aa4bdcf862cdfbf23c9, State: Initialized, Role: FOLLOWER
I20260812 07:03:41.597182 8286 consensus_queue.cc:260] T e7b449f219fa4590b5a97dc7a7376292 P 5e52441d04c64aa4bdcf862cdfbf23c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } }
I20260812 07:03:41.597669 8292 tablet_replica.cc:439] T e7b449f219fa4590b5a97dc7a7376292 P 5e52441d04c64aa4bdcf862cdfbf23c9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } } }
I20260812 07:03:41.598007 8292 tablet_replica.cc:442] T e7b449f219fa4590b5a97dc7a7376292 P 5e52441d04c64aa4bdcf862cdfbf23c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:03:41.599069 8286 ts_tablet_manager.cc:1428] T e7b449f219fa4590b5a97dc7a7376292 P 5e52441d04c64aa4bdcf862cdfbf23c9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 07:03:41.602756 8287 tablet_bootstrap.cc:492] T e7b449f219fa4590b5a97dc7a7376292 P c335f4241d1f4edcb6a58d348f3e7736: No bootstrap required, opened a new log
I20260812 07:03:41.602974 8287 ts_tablet_manager.cc:1397] T e7b449f219fa4590b5a97dc7a7376292 P c335f4241d1f4edcb6a58d348f3e7736: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260812 07:03:41.603724 8287 raft_consensus.cc:348] T e7b449f219fa4590b5a97dc7a7376292 P c335f4241d1f4edcb6a58d348f3e7736 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } }
I20260812 07:03:41.603996 8287 raft_consensus.cc:374] T e7b449f219fa4590b5a97dc7a7376292 P c335f4241d1f4edcb6a58d348f3e7736 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:03:41.604115 8287 raft_consensus.cc:729] T e7b449f219fa4590b5a97dc7a7376292 P c335f4241d1f4edcb6a58d348f3e7736 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c335f4241d1f4edcb6a58d348f3e7736, State: Initialized, Role: FOLLOWER
I20260812 07:03:41.604425 8287 consensus_queue.cc:260] T e7b449f219fa4590b5a97dc7a7376292 P c335f4241d1f4edcb6a58d348f3e7736 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } }
I20260812 07:03:41.605247 8287 ts_tablet_manager.cc:1428] T e7b449f219fa4590b5a97dc7a7376292 P c335f4241d1f4edcb6a58d348f3e7736: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260812 07:03:41.605031 8310 tablet_replica.cc:439] T e7b449f219fa4590b5a97dc7a7376292 P c335f4241d1f4edcb6a58d348f3e7736: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } } }
I20260812 07:03:41.605453 8310 tablet_replica.cc:442] T e7b449f219fa4590b5a97dc7a7376292 P c335f4241d1f4edcb6a58d348f3e7736: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:03:41.624971 8294 raft_consensus.cc:482] T e7b449f219fa4590b5a97dc7a7376292 P fb75cd25c1f045a4a036e627b374310a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:03:41.625321 8294 raft_consensus.cc:504] T e7b449f219fa4590b5a97dc7a7376292 P fb75cd25c1f045a4a036e627b374310a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } }
I20260812 07:03:41.626950 8294 leader_election.cc:288] T e7b449f219fa4590b5a97dc7a7376292 P fb75cd25c1f045a4a036e627b374310a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e52441d04c64aa4bdcf862cdfbf23c9 (127.6.61.1:38735), c335f4241d1f4edcb6a58d348f3e7736 (127.6.61.2:36919)
I20260812 07:03:41.634335 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:41.627786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:52383 (local address 127.6.61.1:38735)
0812 07:03:41.627975 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:41.627981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:41.628001 (+ 20us) server_negotiation.cc:407] Connection header received
0812 07:03:41.628190 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:41.628197 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:41.628335 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:41.628418 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:03:41.629185 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.630163 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.632322 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.633269 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.633415 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:41.633521 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:41.633898 (+ 377us) server_negotiation.cc:299] Negotiation successful
0812 07:03:41.634045 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 07:03:41.635109 8340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:41.627113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:38735 (local address 127.6.61.3:52383)
0812 07:03:41.627848 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:41.627869 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:41.628005 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:41.628683 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:41.628691 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:41.628725 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:03:41.628998 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:41.629008 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.630303 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.630311 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:41.632136 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:41.632149 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.633444 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.633449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:41.633654 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:41.633687 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 07:03:41.634544 (+ 857us) client_negotiation.cc:240] Negotiation successful
0812 07:03:41.634688 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":1,"thread_start_us":888,"threads_started":1}
I20260812 07:03:41.636502 7881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7b449f219fa4590b5a97dc7a7376292" candidate_uuid: "fb75cd25c1f045a4a036e627b374310a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" is_pre_election: true
I20260812 07:03:41.636785 7881 raft_consensus.cc:2454] T e7b449f219fa4590b5a97dc7a7376292 P 5e52441d04c64aa4bdcf862cdfbf23c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb75cd25c1f045a4a036e627b374310a in term 0.
I20260812 07:03:41.637145 8285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:41.627891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:51985 (local address 127.6.61.2:36919)
0812 07:03:41.628079 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:41.628085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:41.628748 (+ 663us) server_negotiation.cc:407] Connection header received
0812 07:03:41.628911 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:41.628917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:41.629020 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:41.629123 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:03:41.630967 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.632149 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.634880 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.636043 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.636205 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:41.636308 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:41.636706 (+ 398us) server_negotiation.cc:299] Negotiation successful
0812 07:03:41.636875 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 07:03:41.637141 8341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:41.627393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:36919 (local address 127.6.61.3:51985)
0812 07:03:41.628638 (+ 1245us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:41.628656 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:41.628750 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:41.630356 (+ 1606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:41.630363 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:41.630394 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:03:41.630785 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:41.630797 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.632585 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.632591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:41.634675 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:41.634692 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.636295 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.636304 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:41.636494 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:41.636528 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 07:03:41.636643 (+ 115us) client_negotiation.cc:240] Negotiation successful
0812 07:03:41.636786 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1109,"mutex_wait_us":27,"thread_start_us":914,"threads_started":1}
I20260812 07:03:41.637669 8125 leader_election.cc:302] T e7b449f219fa4590b5a97dc7a7376292 P fb75cd25c1f045a4a036e627b374310a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e52441d04c64aa4bdcf862cdfbf23c9, fb75cd25c1f045a4a036e627b374310a; no voters:
I20260812 07:03:41.637974 8046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7b449f219fa4590b5a97dc7a7376292" candidate_uuid: "fb75cd25c1f045a4a036e627b374310a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c335f4241d1f4edcb6a58d348f3e7736" is_pre_election: true
I20260812 07:03:41.638180 8294 raft_consensus.cc:2781] T e7b449f219fa4590b5a97dc7a7376292 P fb75cd25c1f045a4a036e627b374310a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:03:41.638273 8046 raft_consensus.cc:2454] T e7b449f219fa4590b5a97dc7a7376292 P c335f4241d1f4edcb6a58d348f3e7736 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb75cd25c1f045a4a036e627b374310a in term 0.
I20260812 07:03:41.638348 8294 raft_consensus.cc:482] T e7b449f219fa4590b5a97dc7a7376292 P fb75cd25c1f045a4a036e627b374310a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:03:41.638470 8294 raft_consensus.cc:3037] T e7b449f219fa4590b5a97dc7a7376292 P fb75cd25c1f045a4a036e627b374310a [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:03:41.641867 8294 raft_consensus.cc:504] T e7b449f219fa4590b5a97dc7a7376292 P fb75cd25c1f045a4a036e627b374310a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } }
I20260812 07:03:41.642665 8294 leader_election.cc:288] T e7b449f219fa4590b5a97dc7a7376292 P fb75cd25c1f045a4a036e627b374310a [CANDIDATE]: Term 1 election: Requested vote from peers 5e52441d04c64aa4bdcf862cdfbf23c9 (127.6.61.1:38735), c335f4241d1f4edcb6a58d348f3e7736 (127.6.61.2:36919)
I20260812 07:03:41.643221 7881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7b449f219fa4590b5a97dc7a7376292" candidate_uuid: "fb75cd25c1f045a4a036e627b374310a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9"
I20260812 07:03:41.643445 7881 raft_consensus.cc:3037] T e7b449f219fa4590b5a97dc7a7376292 P 5e52441d04c64aa4bdcf862cdfbf23c9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:03:41.646636 7881 raft_consensus.cc:2454] T e7b449f219fa4590b5a97dc7a7376292 P 5e52441d04c64aa4bdcf862cdfbf23c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb75cd25c1f045a4a036e627b374310a in term 1.
I20260812 07:03:41.647070 8046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7b449f219fa4590b5a97dc7a7376292" candidate_uuid: "fb75cd25c1f045a4a036e627b374310a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c335f4241d1f4edcb6a58d348f3e7736"
I20260812 07:03:41.647154 8125 leader_election.cc:302] T e7b449f219fa4590b5a97dc7a7376292 P fb75cd25c1f045a4a036e627b374310a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e52441d04c64aa4bdcf862cdfbf23c9, fb75cd25c1f045a4a036e627b374310a; no voters:
I20260812 07:03:41.647352 8046 raft_consensus.cc:3037] T e7b449f219fa4590b5a97dc7a7376292 P c335f4241d1f4edcb6a58d348f3e7736 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:03:41.647468 8294 raft_consensus.cc:2781] T e7b449f219fa4590b5a97dc7a7376292 P fb75cd25c1f045a4a036e627b374310a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:03:41.650645 8046 raft_consensus.cc:2454] T e7b449f219fa4590b5a97dc7a7376292 P c335f4241d1f4edcb6a58d348f3e7736 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb75cd25c1f045a4a036e627b374310a in term 1.
I20260812 07:03:41.651115 8294 raft_consensus.cc:686] T e7b449f219fa4590b5a97dc7a7376292 P fb75cd25c1f045a4a036e627b374310a [term 1 LEADER]: Becoming Leader. State: Replica: fb75cd25c1f045a4a036e627b374310a, State: Running, Role: LEADER
I20260812 07:03:41.651561 8294 consensus_queue.cc:237] T e7b449f219fa4590b5a97dc7a7376292 P fb75cd25c1f045a4a036e627b374310a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } }
I20260812 07:03:41.653910 8342 tablet_replica.cc:439] T e7b449f219fa4590b5a97dc7a7376292 P fb75cd25c1f045a4a036e627b374310a: TxnStatusTablet state changed. Reason: New leader fb75cd25c1f045a4a036e627b374310a. Latest consensus state: current_term: 1 leader_uuid: "fb75cd25c1f045a4a036e627b374310a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } } }
I20260812 07:03:41.654331 8342 tablet_replica.cc:442] T e7b449f219fa4590b5a97dc7a7376292 P fb75cd25c1f045a4a036e627b374310a: This TxnStatusTablet replica's current role is: LEADER
I20260812 07:03:41.655808 7682 catalog_manager.cc:5495] T e7b449f219fa4590b5a97dc7a7376292 P fb75cd25c1f045a4a036e627b374310a reported cstate change: term changed from 0 to 1, leader changed from <none> to fb75cd25c1f045a4a036e627b374310a (127.6.61.3). New cstate: current_term: 1 leader_uuid: "fb75cd25c1f045a4a036e627b374310a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } health_report { overall_health: HEALTHY } } }
I20260812 07:03:41.656205 8344 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 07:03:41.656558 8344 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 07:03:41.691439 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:41.683615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46440 (local address 127.6.61.3:35519)
0812 07:03:41.683824 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:41.683830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:41.683980 (+ 150us) server_negotiation.cc:407] Connection header received
0812 07:03:41.684306 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:41.684312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:41.684423 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:41.684609 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:03:41.685055 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.686335 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.688355 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.689135 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.689294 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:41.689925 (+ 631us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:03:41.690675 (+ 750us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:03:41.690829 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:41.690982 (+ 153us) server_negotiation.cc:299] Negotiation successful
0812 07:03:41.691079 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 07:03:41.691797 8333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:41.683471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:35519 (local address 127.0.0.1:46440)
0812 07:03:41.683870 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:41.683892 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:41.684058 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:41.684652 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:41.684658 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:41.684672 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 07:03:41.684885 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:41.684899 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.686490 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.686497 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:41.688177 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:41.688193 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.689331 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.689337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:41.689563 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:41.689649 (+ 86us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 07:03:41.690810 (+ 1161us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 07:03:41.690837 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 07:03:41.690937 (+ 100us) client_negotiation.cc:240] Negotiation successful
0812 07:03:41.691193 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229}
I20260812 07:03:41.696461 7881 raft_consensus.cc:1261] T e7b449f219fa4590b5a97dc7a7376292 P 5e52441d04c64aa4bdcf862cdfbf23c9 [term 1 FOLLOWER]: Refusing update from remote peer fb75cd25c1f045a4a036e627b374310a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:03:41.696630 8046 raft_consensus.cc:1261] T e7b449f219fa4590b5a97dc7a7376292 P c335f4241d1f4edcb6a58d348f3e7736 [term 1 FOLLOWER]: Refusing update from remote peer fb75cd25c1f045a4a036e627b374310a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:03:41.697412 8342 consensus_queue.cc:1035] T e7b449f219fa4590b5a97dc7a7376292 P fb75cd25c1f045a4a036e627b374310a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:03:41.701596 8294 consensus_queue.cc:1035] T e7b449f219fa4590b5a97dc7a7376292 P fb75cd25c1f045a4a036e627b374310a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:03:41.714103 8292 tablet_replica.cc:439] T e7b449f219fa4590b5a97dc7a7376292 P 5e52441d04c64aa4bdcf862cdfbf23c9: TxnStatusTablet state changed. Reason: New leader fb75cd25c1f045a4a036e627b374310a. Latest consensus state: current_term: 1 leader_uuid: "fb75cd25c1f045a4a036e627b374310a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } } }
I20260812 07:03:41.714540 8292 tablet_replica.cc:442] T e7b449f219fa4590b5a97dc7a7376292 P 5e52441d04c64aa4bdcf862cdfbf23c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:03:41.717372 8304 mvcc.cc:205] Tried to move back new op lower bound from 7317578636060872704 to 7317578635888484352. Current Snapshot: MvccSnapshot[applied={T|T < 7317578636060872704}]
I20260812 07:03:41.718241 8305 mvcc.cc:205] Tried to move back new op lower bound from 7317578636060872704 to 7317578635888484352. Current Snapshot: MvccSnapshot[applied={T|T < 7317578636060872704 or (T in {7317578636060872704})}]
I20260812 07:03:41.718705 8292 tablet_replica.cc:439] T e7b449f219fa4590b5a97dc7a7376292 P 5e52441d04c64aa4bdcf862cdfbf23c9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb75cd25c1f045a4a036e627b374310a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } } }
I20260812 07:03:41.719051 8292 tablet_replica.cc:442] T e7b449f219fa4590b5a97dc7a7376292 P 5e52441d04c64aa4bdcf862cdfbf23c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:03:41.726611 8310 tablet_replica.cc:439] T e7b449f219fa4590b5a97dc7a7376292 P c335f4241d1f4edcb6a58d348f3e7736: TxnStatusTablet state changed. Reason: New leader fb75cd25c1f045a4a036e627b374310a. Latest consensus state: current_term: 1 leader_uuid: "fb75cd25c1f045a4a036e627b374310a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } } }
I20260812 07:03:41.726958 8310 tablet_replica.cc:442] T e7b449f219fa4590b5a97dc7a7376292 P c335f4241d1f4edcb6a58d348f3e7736: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:03:41.734316 8310 tablet_replica.cc:439] T e7b449f219fa4590b5a97dc7a7376292 P c335f4241d1f4edcb6a58d348f3e7736: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb75cd25c1f045a4a036e627b374310a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } } }
I20260812 07:03:41.734591 8310 tablet_replica.cc:442] T e7b449f219fa4590b5a97dc7a7376292 P c335f4241d1f4edcb6a58d348f3e7736: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:03:41.736860 8294 tablet_replica.cc:439] T e7b449f219fa4590b5a97dc7a7376292 P fb75cd25c1f045a4a036e627b374310a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fb75cd25c1f045a4a036e627b374310a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } } }
I20260812 07:03:41.737192 8294 tablet_replica.cc:442] T e7b449f219fa4590b5a97dc7a7376292 P fb75cd25c1f045a4a036e627b374310a: This TxnStatusTablet replica's current role is: LEADER
I20260812 07:03:41.738195 8342 tablet_replica.cc:439] T e7b449f219fa4590b5a97dc7a7376292 P fb75cd25c1f045a4a036e627b374310a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fb75cd25c1f045a4a036e627b374310a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e52441d04c64aa4bdcf862cdfbf23c9" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 38735 } } peers { permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 } } peers { permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 } } }
I20260812 07:03:41.738436 8342 tablet_replica.cc:442] T e7b449f219fa4590b5a97dc7a7376292 P fb75cd25c1f045a4a036e627b374310a: This TxnStatusTablet replica's current role is: LEADER
I20260812 07:03:41.748922 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:41.739938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53768 (local address 127.6.61.1:38735)
0812 07:03:41.740118 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:41.740124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:41.741934 (+ 1810us) server_negotiation.cc:407] Connection header received
0812 07:03:41.742177 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:41.742183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:41.742308 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:41.742477 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:03:41.743063 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.744187 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.746156 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.746841 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.746976 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:41.747485 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:03:41.748206 (+ 721us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:03:41.748348 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:41.748498 (+ 150us) server_negotiation.cc:299] Negotiation successful
0812 07:03:41.748590 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 07:03:41.770668 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:41.762957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:33483 (local address 127.6.61.3:35519)
0812 07:03:41.763137 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:41.763143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:41.763753 (+ 610us) server_negotiation.cc:407] Connection header received
0812 07:03:41.764352 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:41.764358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:41.764450 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:41.764559 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:03:41.765095 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.766286 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.768381 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.769143 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.769286 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:41.769856 (+ 570us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:03:41.769996 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:03:41.770083 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:41.770246 (+ 163us) server_negotiation.cc:299] Negotiation successful
0812 07:03:41.770339 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 07:03:41.770815 8356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:41.762914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:35519 (local address 127.6.61.1:33483)
0812 07:03:41.763618 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:41.763637 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:41.764184 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:41.764590 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:41.764596 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:41.764610 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 07:03:41.764940 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:41.764951 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.766444 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.766450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:41.768226 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:41.768241 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.769370 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.769379 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:41.769562 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:41.769634 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 07:03:41.770110 (+ 476us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 07:03:41.770122 (+ 12us) client_negotiation.cc:769] Sending connection context
0812 07:03:41.770197 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 07:03:41.770344 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":375,"threads_started":1}
I20260812 07:03:41.784919 8214 raft_consensus.cc:1261] T 5a5f3b50042c4bcbb7a93149b83c21b7 P fb75cd25c1f045a4a036e627b374310a [term 1 FOLLOWER]: Refusing update from remote peer 5e52441d04c64aa4bdcf862cdfbf23c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:03:41.785074 8046 raft_consensus.cc:1261] T 5a5f3b50042c4bcbb7a93149b83c21b7 P c335f4241d1f4edcb6a58d348f3e7736 [term 1 FOLLOWER]: Refusing update from remote peer 5e52441d04c64aa4bdcf862cdfbf23c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:03:41.785804 8324 consensus_queue.cc:1035] T 5a5f3b50042c4bcbb7a93149b83c21b7 P 5e52441d04c64aa4bdcf862cdfbf23c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c335f4241d1f4edcb6a58d348f3e7736" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:03:41.786808 8292 consensus_queue.cc:1035] T 5a5f3b50042c4bcbb7a93149b83c21b7 P 5e52441d04c64aa4bdcf862cdfbf23c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb75cd25c1f045a4a036e627b374310a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:03:41.849341 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:41.839726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:60901 (local address 127.6.61.1:38735)
0812 07:03:41.839899 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:41.839906 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:41.842090 (+ 2184us) server_negotiation.cc:407] Connection header received
0812 07:03:41.842297 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:41.842303 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:41.842385 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:41.842484 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:03:41.843079 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.844224 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.846970 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.847699 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.847792 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:41.848466 (+ 674us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:03:41.848623 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:03:41.848704 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:41.848935 (+ 231us) server_negotiation.cc:299] Negotiation successful
0812 07:03:41.849019 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 07:03:41.850176 8361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:41.840059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:38735 (local address 127.6.61.3:60901)
0812 07:03:41.841975 (+ 1916us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:41.841996 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:41.842152 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:41.842521 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:41.842527 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:41.842541 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 07:03:41.842909 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:41.842923 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.844452 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.844863 (+ 411us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:41.846777 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:41.846793 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.847994 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.848000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:41.848171 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:41.848247 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 07:03:41.848797 (+ 550us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 07:03:41.848810 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 07:03:41.849580 (+ 770us) client_negotiation.cc:240] Negotiation successful
0812 07:03:41.849746 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1682,"thread_start_us":1501,"threads_started":1}
I20260812 07:03:41.889984 7682 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:03:41.890357 7682 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:03:41.894817 7682 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f1dea41f34141ffa6efc53e55582c43: Sending DeleteTablet for 3 replicas of tablet 5a5f3b50042c4bcbb7a93149b83c21b7
I20260812 07:03:41.895948 8019 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a5f3b50042c4bcbb7a93149b83c21b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:03:41 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:33038
I20260812 07:03:41.900795 7852 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a5f3b50042c4bcbb7a93149b83c21b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:03:41 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:53766
I20260812 07:03:41.904481 8181 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a5f3b50042c4bcbb7a93149b83c21b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:03:41 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:46428
I20260812 07:03:41.906262 8366 tablet_replica.cc:330] T 5a5f3b50042c4bcbb7a93149b83c21b7 P 5e52441d04c64aa4bdcf862cdfbf23c9: stopping tablet replica
I20260812 07:03:41.906482 8366 raft_consensus.cc:2229] T 5a5f3b50042c4bcbb7a93149b83c21b7 P 5e52441d04c64aa4bdcf862cdfbf23c9 [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:03:41.907104 8366 raft_consensus.cc:2258] T 5a5f3b50042c4bcbb7a93149b83c21b7 P 5e52441d04c64aa4bdcf862cdfbf23c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:03:41.908313 8366 ts_tablet_manager.cc:1905] T 5a5f3b50042c4bcbb7a93149b83c21b7 P 5e52441d04c64aa4bdcf862cdfbf23c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:03:41.914381 8367 tablet_replica.cc:330] T 5a5f3b50042c4bcbb7a93149b83c21b7 P fb75cd25c1f045a4a036e627b374310a: stopping tablet replica
I20260812 07:03:41.914604 8367 raft_consensus.cc:2229] T 5a5f3b50042c4bcbb7a93149b83c21b7 P fb75cd25c1f045a4a036e627b374310a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:03:41.914934 8367 raft_consensus.cc:2258] T 5a5f3b50042c4bcbb7a93149b83c21b7 P fb75cd25c1f045a4a036e627b374310a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:03:41.918402 8364 tablet_replica.cc:330] T 5a5f3b50042c4bcbb7a93149b83c21b7 P c335f4241d1f4edcb6a58d348f3e7736: stopping tablet replica
I20260812 07:03:41.918815 8364 raft_consensus.cc:2229] T 5a5f3b50042c4bcbb7a93149b83c21b7 P c335f4241d1f4edcb6a58d348f3e7736 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:03:41.919160 8364 raft_consensus.cc:2258] T 5a5f3b50042c4bcbb7a93149b83c21b7 P c335f4241d1f4edcb6a58d348f3e7736 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:03:41.920554 8364 ts_tablet_manager.cc:1905] T 5a5f3b50042c4bcbb7a93149b83c21b7 P c335f4241d1f4edcb6a58d348f3e7736: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:03:41.920812 8367 ts_tablet_manager.cc:1905] T 5a5f3b50042c4bcbb7a93149b83c21b7 P fb75cd25c1f045a4a036e627b374310a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:03:41.928280 8367 ts_tablet_manager.cc:1918] T 5a5f3b50042c4bcbb7a93149b83c21b7 P fb75cd25c1f045a4a036e627b374310a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 07:03:41.928443 8367 log.cc:1196] T 5a5f3b50042c4bcbb7a93149b83c21b7 P fb75cd25c1f045a4a036e627b374310a: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-2/wal/wals/5a5f3b50042c4bcbb7a93149b83c21b7
I20260812 07:03:41.929679 8364 ts_tablet_manager.cc:1918] T 5a5f3b50042c4bcbb7a93149b83c21b7 P c335f4241d1f4edcb6a58d348f3e7736: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 07:03:41.929963 8364 log.cc:1196] T 5a5f3b50042c4bcbb7a93149b83c21b7 P c335f4241d1f4edcb6a58d348f3e7736: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-1/wal/wals/5a5f3b50042c4bcbb7a93149b83c21b7
I20260812 07:03:41.930883 8364 ts_tablet_manager.cc:1939] T 5a5f3b50042c4bcbb7a93149b83c21b7 P c335f4241d1f4edcb6a58d348f3e7736: Deleting consensus metadata
I20260812 07:03:41.932781 7667 catalog_manager.cc:4841] TS c335f4241d1f4edcb6a58d348f3e7736 (127.6.61.2:36919): tablet 5a5f3b50042c4bcbb7a93149b83c21b7 (table test-table [id=1e2eff09f95846158facbde0a926ff30]) successfully deleted
I20260812 07:03:41.936760 8366 ts_tablet_manager.cc:1918] T 5a5f3b50042c4bcbb7a93149b83c21b7 P 5e52441d04c64aa4bdcf862cdfbf23c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 07:03:41.937077 8366 log.cc:1196] T 5a5f3b50042c4bcbb7a93149b83c21b7 P 5e52441d04c64aa4bdcf862cdfbf23c9: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518208686283-6388-0/minicluster-data/ts-0/wal/wals/5a5f3b50042c4bcbb7a93149b83c21b7
I20260812 07:03:41.937603 8366 ts_tablet_manager.cc:1939] T 5a5f3b50042c4bcbb7a93149b83c21b7 P 5e52441d04c64aa4bdcf862cdfbf23c9: Deleting consensus metadata
I20260812 07:03:41.938834 7666 catalog_manager.cc:4841] TS 5e52441d04c64aa4bdcf862cdfbf23c9 (127.6.61.1:38735): tablet 5a5f3b50042c4bcbb7a93149b83c21b7 (table test-table [id=1e2eff09f95846158facbde0a926ff30]) successfully deleted
I20260812 07:03:41.939540 8367 ts_tablet_manager.cc:1939] T 5a5f3b50042c4bcbb7a93149b83c21b7 P fb75cd25c1f045a4a036e627b374310a: Deleting consensus metadata
I20260812 07:03:41.940737 7668 catalog_manager.cc:4841] TS fb75cd25c1f045a4a036e627b374310a (127.6.61.3:35519): tablet 5a5f3b50042c4bcbb7a93149b83c21b7 (table test-table [id=1e2eff09f95846158facbde0a926ff30]) successfully deleted
Aug 12 07:03:41 dist-test-slave-69rg krb5kdc[7634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518220, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.61.1@KRBTEST.COM
I20260812 07:03:41.960589 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:41.934909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53778 (local address 127.6.61.1:38735)
0812 07:03:41.935108 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:41.935115 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:41.941657 (+ 6542us) server_negotiation.cc:407] Connection header received
0812 07:03:41.941991 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:41.941998 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:41.942074 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:41.942139 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:41.944261 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.945446 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.947364 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.948037 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.948132 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:41.953608 (+ 5476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:41.953625 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:41.953644 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:41.953689 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:41.957456 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:41.958480 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:41.958490 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:41.958494 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:41.958644 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:41.959044 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:41.959049 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:41.959053 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:41.959410 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:41.959737 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:41.959992 (+ 255us) server_negotiation.cc:299] Negotiation successful
0812 07:03:41.960213 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260812 07:03:41.961350 7929 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53778
I20260812 07:03:41.987728 8334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:41.971195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38166 (local address 127.6.61.62:44745)
0812 07:03:41.971455 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:41.971462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:41.974446 (+ 2984us) server_negotiation.cc:407] Connection header received
0812 07:03:41.974673 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:41.974679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:41.974813 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:41.974935 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:41.976549 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.977439 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.978797 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.979250 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.979404 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:41.981985 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:41.981990 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:41.981993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:41.982024 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:41.984680 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:41.985548 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:41.985553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:41.985557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:41.985691 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:41.986135 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:41.986141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:41.986144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:41.986549 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:41.986811 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:41.987167 (+ 356us) server_negotiation.cc:299] Negotiation successful
0812 07:03:41.987420 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260812 07:03:41.988582 7682 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38166
I20260812 07:03:41.992607 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 7769
I20260812 07:03:42.007714 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:41.993761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53790 (local address 127.6.61.1:38735)
0812 07:03:41.994054 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:41.994067 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:41.994250 (+ 183us) server_negotiation.cc:407] Connection header received
0812 07:03:41.994504 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:41.994509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:41.994588 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:41.994698 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:41.996025 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.997077 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.998750 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:41.999459 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:41.999607 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:42.001884 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:42.001890 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:42.001895 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:42.001937 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:42.004863 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:42.005510 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:42.005517 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:42.005521 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:42.005693 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:42.006124 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:42.006129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:42.006133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:42.006416 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:42.006701 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:42.007042 (+ 341us) server_negotiation.cc:299] Negotiation successful
0812 07:03:42.007293 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
W20260812 07:03:42.008316 7929 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53790
W20260812 07:03:42.008920 6388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 7769: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 07:03:42.009131 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 7769
I20260812 07:03:42.033792 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 7940
Aug 12 07:03:42 dist-test-slave-69rg krb5kdc[7634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518220, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.61.2@KRBTEST.COM
I20260812 07:03:42.049522 8285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:42.034646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33054 (local address 127.6.61.2:36919)
0812 07:03:42.034855 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:42.034863 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:42.034885 (+ 22us) server_negotiation.cc:407] Connection header received
0812 07:03:42.035178 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:42.035182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:42.035252 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:42.035366 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:42.036545 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:42.037270 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:42.038711 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:42.039139 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:42.039278 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:42.043348 (+ 4070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:42.043363 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:42.043378 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:42.043414 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:42.046696 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:42.047382 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:42.047398 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:42.047400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:42.047532 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:42.047924 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:42.047929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:42.047932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:42.048292 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:42.048638 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:42.048929 (+ 291us) server_negotiation.cc:299] Negotiation successful
0812 07:03:42.049174 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260812 07:03:42.050352 8094 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33054
W20260812 07:03:42.051116 6388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 7940: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 07:03:42.051332 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 7940
I20260812 07:03:42.089356 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 8105
Aug 12 07:03:42 dist-test-slave-69rg krb5kdc[7634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518220, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.61.3@KRBTEST.COM
I20260812 07:03:42.107352 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:42.090286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46452 (local address 127.6.61.3:35519)
0812 07:03:42.090475 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:42.090482 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:42.090505 (+ 23us) server_negotiation.cc:407] Connection header received
0812 07:03:42.090699 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:42.090705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:42.090792 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:42.090902 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:42.092174 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:42.093068 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:42.094480 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:42.094918 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:42.095085 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:42.100202 (+ 5117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:42.100216 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:42.100233 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:42.100275 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:42.104043 (+ 3768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:42.104893 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:42.104901 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:42.104903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:42.105069 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:42.105532 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:42.105537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:42.105541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:42.106018 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:42.106384 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:42.106702 (+ 318us) server_negotiation.cc:299] Negotiation successful
0812 07:03:42.106949 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260812 07:03:42.108383 8256 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46452
W20260812 07:03:42.109067 6388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 8105: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 07:03:42.109225 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 8105
I20260812 07:03:42.134897 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 7649
I20260812 07:03:42.136137 7733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:03:42.253207 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 7649
2026-08-12T07:03:42Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4935 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:03:42 dist-test-slave-69rg krb5kdc[8377](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:03:42 dist-test-slave-69rg krb5kdc[8377](info): set up 2 sockets
Aug 12 07:03:42 dist-test-slave-69rg krb5kdc[8377](info): commencing operation
krb5kdc: starting...
W20260812 07:03:43.436659 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:03:43 dist-test-slave-69rg krb5kdc[8377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:03:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:03:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
I20260812 07:03:43.653772 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:39233
--webserver_interface=127.6.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:39233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:03:43.981213 8392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:03:43.981500 8392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:03:43.981591 8392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:03:43.988844 8392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:03:43.988961 8392 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 07:03:43.989037 8392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:03:43.989132 8392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:03:43.989226 8392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:03:43.989320 8392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:03:43.999413 8392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:39233
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:39233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:03:44.000743 8392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:03:44.002172 8392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:03:44.014407 8397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:44.014324 8398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:44.014346 8400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:03:44.016008 8392 server_base.cc:1034] running on GCE node
I20260812 07:03:44.016773 8392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:03:44.018219 8392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:03:44.019483 8392 hybrid_clock.cc:648] HybridClock initialized: now 1786518224019423 us; error 35 us; skew 500 ppm
I20260812 07:03:44.019841 8392 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:03:44 dist-test-slave-69rg krb5kdc[8377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518224, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:03:44.023025 8392 init.cc:377] Logged in from keytab as kudu/127.6.61.62@KRBTEST.COM (short username kudu)
I20260812 07:03:44.024525 8392 webserver.cc:460] Webserver started at http://127.6.61.62:44919/ using document root <none> and password file <none>
I20260812 07:03:44.025173 8392 fs_manager.cc:359] Metadata directory not provided
I20260812 07:03:44.025264 8392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:03:44.025529 8392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:03:44.028475 8392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "100f02a2c2c24a9f9f301631ca439064"
format_stamp: "Formatted at 2026-08-12 07:03:44 on dist-test-slave-69rg"
server_key: "a2e172690db62c4fc52ab9dbe375a020"
server_key_iv: "57a32738751824c38cd0adb5c94be216"
server_key_version: "encryptionkey@0"
I20260812 07:03:44.029260 8392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "100f02a2c2c24a9f9f301631ca439064"
format_stamp: "Formatted at 2026-08-12 07:03:44 on dist-test-slave-69rg"
server_key: "a2e172690db62c4fc52ab9dbe375a020"
server_key_iv: "57a32738751824c38cd0adb5c94be216"
server_key_version: "encryptionkey@0"
I20260812 07:03:44.034278 8392 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 07:03:44.037590 8407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:44.039021 8392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:03:44.039209 8392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "100f02a2c2c24a9f9f301631ca439064"
format_stamp: "Formatted at 2026-08-12 07:03:44 on dist-test-slave-69rg"
server_key: "a2e172690db62c4fc52ab9dbe375a020"
server_key_iv: "57a32738751824c38cd0adb5c94be216"
server_key_version: "encryptionkey@0"
I20260812 07:03:44.039436 8392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:03:44.072696 8392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:03:44.076938 8392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:03:44.077215 8392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:03:44.099563 8392 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:39233
I20260812 07:03:44.099583 8479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:39233 every 8 connection(s)
I20260812 07:03:44.101046 8392 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:03:44.104269 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 8392
I20260812 07:03:44.104506 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:03:44.105373 6388 external_mini_cluster.cc:1428] Setting key 88cb5843279c0665ef0093f1c95f8a0a
I20260812 07:03:44.105620 8480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:03:44.120554 8480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 100f02a2c2c24a9f9f301631ca439064: Bootstrap starting.
Aug 12 07:03:44 dist-test-slave-69rg krb5kdc[8377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:03:44.123883 8480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 100f02a2c2c24a9f9f301631ca439064: Neither blocks nor log segments found. Creating new log.
I20260812 07:03:44.125007 8480 log.cc:824] T 00000000000000000000000000000000 P 100f02a2c2c24a9f9f301631ca439064: Log is configured to *not* fsync() on all Append() calls
I20260812 07:03:44.127929 8480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 100f02a2c2c24a9f9f301631ca439064: No bootstrap required, opened a new log
I20260812 07:03:44.129365 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:44.107576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54314 (local address 127.6.61.62:39233)
0812 07:03:44.108408 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:44.108419 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:44.108462 (+ 43us) server_negotiation.cc:407] Connection header received
0812 07:03:44.110496 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:44.110520 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:44.110895 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:44.111217 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:44.113039 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:44.114428 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:44.116209 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:44.116901 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:44.117033 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:44.121893 (+ 4860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:44.121907 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:44.121927 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:44.121972 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:44.125128 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:44.125892 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:44.125906 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:44.125913 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:44.126091 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:44.126486 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:44.126491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:44.126495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:44.127018 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:44.127320 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:44.127778 (+ 458us) server_negotiation.cc:299] Negotiation successful
0812 07:03:44.128359 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":387,"threads_started":1}
I20260812 07:03:44.135630 8480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 100f02a2c2c24a9f9f301631ca439064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100f02a2c2c24a9f9f301631ca439064" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39233 } }
I20260812 07:03:44.136024 8480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 100f02a2c2c24a9f9f301631ca439064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:03:44.136139 8480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 100f02a2c2c24a9f9f301631ca439064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 100f02a2c2c24a9f9f301631ca439064, State: Initialized, Role: FOLLOWER
I20260812 07:03:44.136703 8480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 100f02a2c2c24a9f9f301631ca439064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100f02a2c2c24a9f9f301631ca439064" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39233 } }
I20260812 07:03:44.136956 8480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 100f02a2c2c24a9f9f301631ca439064 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:03:44.137107 8480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 100f02a2c2c24a9f9f301631ca439064 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:03:44.137295 8480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 100f02a2c2c24a9f9f301631ca439064 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:03:44.139633 8480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 100f02a2c2c24a9f9f301631ca439064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100f02a2c2c24a9f9f301631ca439064" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39233 } }
I20260812 07:03:44.140199 8480 leader_election.cc:302] T 00000000000000000000000000000000 P 100f02a2c2c24a9f9f301631ca439064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 100f02a2c2c24a9f9f301631ca439064; no voters:
I20260812 07:03:44.140815 8480 leader_election.cc:288] T 00000000000000000000000000000000 P 100f02a2c2c24a9f9f301631ca439064 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:03:44.140980 8485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 100f02a2c2c24a9f9f301631ca439064 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:03:44.141872 8485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 100f02a2c2c24a9f9f301631ca439064 [term 1 LEADER]: Becoming Leader. State: Replica: 100f02a2c2c24a9f9f301631ca439064, State: Running, Role: LEADER
I20260812 07:03:44.142640 8485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 100f02a2c2c24a9f9f301631ca439064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100f02a2c2c24a9f9f301631ca439064" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39233 } }
I20260812 07:03:44.143308 8480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 100f02a2c2c24a9f9f301631ca439064 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:03:44.145522 8487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 100f02a2c2c24a9f9f301631ca439064 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 100f02a2c2c24a9f9f301631ca439064. Latest consensus state: current_term: 1 leader_uuid: "100f02a2c2c24a9f9f301631ca439064" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100f02a2c2c24a9f9f301631ca439064" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39233 } } }
I20260812 07:03:44.145931 8487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 100f02a2c2c24a9f9f301631ca439064 [sys.catalog]: This master's current role is: LEADER
I20260812 07:03:44.146538 8486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 100f02a2c2c24a9f9f301631ca439064 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "100f02a2c2c24a9f9f301631ca439064" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100f02a2c2c24a9f9f301631ca439064" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39233 } } }
I20260812 07:03:44.146826 8486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 100f02a2c2c24a9f9f301631ca439064 [sys.catalog]: This master's current role is: LEADER
I20260812 07:03:44.150969 8492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:03:44.156364 8492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:03:44.166190 8492 catalog_manager.cc:1347] Generated new cluster ID: 218ae54d4d8a4fa28ad3195190877058
I20260812 07:03:44.166330 8492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:03:44.174142 8492 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:03:44.175923 8492 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:03:44.193331 8492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 100f02a2c2c24a9f9f301631ca439064: Generated new TSK 0
I20260812 07:03:44.194316 8492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
I20260812 07:03:44.261955 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.1:0
--local_ip_for_outbound_sockets=127.6.61.1
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:39233
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:03:44.594750 8511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:03:44.595093 8511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:03:44.595288 8511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:03:44.602819 8511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:03:44.602986 8511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:03:44.603170 8511 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 07:03:44.603227 8511 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 07:03:44.603300 8511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.1
I20260812 07:03:44.613270 8511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.61.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.61.62:39233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.1
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:03:44.614586 8511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:03:44.615886 8511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:03:44.630056 8516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:44.630100 8518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:44.630143 8520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:03:44.631896 8511 server_base.cc:1034] running on GCE node
I20260812 07:03:44.632575 8511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:03:44.633548 8511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:03:44.634752 8511 hybrid_clock.cc:648] HybridClock initialized: now 1786518224634691 us; error 54 us; skew 500 ppm
I20260812 07:03:44.635082 8511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:03:44 dist-test-slave-69rg krb5kdc[8377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518224, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:03:44.638337 8511 init.cc:377] Logged in from keytab as kudu/127.6.61.1@KRBTEST.COM (short username kudu)
I20260812 07:03:44.639787 8511 webserver.cc:460] Webserver started at http://127.6.61.1:34111/ using document root <none> and password file <none>
I20260812 07:03:44.640511 8511 fs_manager.cc:359] Metadata directory not provided
I20260812 07:03:44.640646 8511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:03:44.640981 8511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:03:44.644222 8511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-0/data/instance:
uuid: "a6e3ffc000c840d1b0f8d22e592b556b"
format_stamp: "Formatted at 2026-08-12 07:03:44 on dist-test-slave-69rg"
server_key: "f97c6d781e118ca95f4ab3c5f97519fc"
server_key_iv: "6b79956a4d0a547edf5f369cc6278f2b"
server_key_version: "encryptionkey@0"
I20260812 07:03:44.645020 8511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance:
uuid: "a6e3ffc000c840d1b0f8d22e592b556b"
format_stamp: "Formatted at 2026-08-12 07:03:44 on dist-test-slave-69rg"
server_key: "f97c6d781e118ca95f4ab3c5f97519fc"
server_key_iv: "6b79956a4d0a547edf5f369cc6278f2b"
server_key_version: "encryptionkey@0"
I20260812 07:03:44.650450 8511 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260812 07:03:44.654222 8528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:44.655592 8511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 07:03:44.655767 8511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-0/wal
uuid: "a6e3ffc000c840d1b0f8d22e592b556b"
format_stamp: "Formatted at 2026-08-12 07:03:44 on dist-test-slave-69rg"
server_key: "f97c6d781e118ca95f4ab3c5f97519fc"
server_key_iv: "6b79956a4d0a547edf5f369cc6278f2b"
server_key_version: "encryptionkey@0"
I20260812 07:03:44.655980 8511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:03:44.694044 8511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:03:44.698385 8511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:03:44.698697 8511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:03:44.701552 8511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:03:44.701723 8511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:44.701897 8511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:03:44.702000 8511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 07:03:44 dist-test-slave-69rg krb5kdc[8377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518224, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:03:44.732137 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:44.702518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:48635 (local address 127.6.61.62:39233)
0812 07:03:44.703148 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:44.703157 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:44.704350 (+ 1193us) server_negotiation.cc:407] Connection header received
0812 07:03:44.707600 (+ 3250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:44.707607 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:44.707706 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:44.707857 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:44.710371 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:44.711427 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:44.714062 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:44.714578 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:44.714756 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:44.724338 (+ 9582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:44.724348 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:44.724355 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:44.724414 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:44.727742 (+ 3328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:44.728934 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:44.728941 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:44.728947 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:44.729111 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:44.729750 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:44.729759 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:44.729764 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:44.730249 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:44.730450 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:44.731228 (+ 778us) server_negotiation.cc:299] Negotiation successful
0812 07:03:44.731559 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":378,"threads_started":1}
I20260812 07:03:44.732731 8540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:44.703212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39233 (local address 127.6.61.1:48635)
0812 07:03:44.704154 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:44.704219 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:44.707142 (+ 2923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:44.708138 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:44.708150 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:44.709041 (+ 891us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:03:44.710100 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:44.710128 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:44.711660 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:44.711669 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:44.713898 (+ 2229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:44.713911 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:44.714938 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:44.714949 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:44.715299 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:44.720526 (+ 5227us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:03:44.720565 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:03:44.723979 (+ 3414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:03:44.727972 (+ 3993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:44.727981 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:44.728000 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:44.728710 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:44.729219 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:44.729226 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:44.729230 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:44.729551 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:44.730410 (+ 859us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:03:44.730417 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:03:44.730790 (+ 373us) client_negotiation.cc:769] Sending connection context
0812 07:03:44.731031 (+ 241us) client_negotiation.cc:240] Negotiation successful
0812 07:03:44.731457 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":413,"threads_started":1}
I20260812 07:03:44.750272 8511 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.1:39565
I20260812 07:03:44.750284 8667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.1:39565 every 8 connection(s)
I20260812 07:03:44.752516 8511 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
I20260812 07:03:44.755314 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 8511
I20260812 07:03:44.755537 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance
I20260812 07:03:44.756295 6388 external_mini_cluster.cc:1428] Setting key d3564752343ba683756099efd35f33d6
I20260812 07:03:44.756984 8669 heartbeater.cc:347] Connected to a master server at 127.6.61.62:39233
I20260812 07:03:44.757267 8669 heartbeater.cc:464] Registering TS with master...
I20260812 07:03:44.758275 8669 heartbeater.cc:511] Master 127.6.61.62:39233 requested a full tablet report, sending...
I20260812 07:03:44.760780 8428 ts_manager.cc:194] Registered new tserver with Master: a6e3ffc000c840d1b0f8d22e592b556b (127.6.61.1:39565)
I20260812 07:03:44.763039 8428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.1@KRBTEST.COM'} at 127.6.61.1:48635
I20260812 07:03:44.767483 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:44.739682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:53867 (local address 127.6.61.62:39233)
0812 07:03:44.739955 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:44.739964 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:44.754522 (+ 14558us) server_negotiation.cc:407] Connection header received
0812 07:03:44.754666 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:44.754673 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:44.754826 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:44.754954 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:44.756059 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:44.757137 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:44.758817 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:44.759405 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:44.759527 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:44.761701 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:44.761710 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:44.761713 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:44.761744 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:44.764761 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:44.765448 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:44.765455 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:44.765458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:44.765585 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:44.766036 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:44.766041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:44.766045 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:44.766323 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:44.766480 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:44.766906 (+ 426us) server_negotiation.cc:299] Negotiation successful
0812 07:03:44.767119 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260812 07:03:44.767544 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:44.739703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39233 (local address 127.6.61.1:53867)
0812 07:03:44.754408 (+ 14705us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:44.754426 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:44.754548 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:44.755006 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:44.755012 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:44.755577 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:03:44.755899 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:44.755910 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:44.757305 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:44.757311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:44.758676 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:44.758690 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:44.759773 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:44.759782 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:44.759924 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:44.760634 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:03:44.760662 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:03:44.761489 (+ 827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:03:44.764892 (+ 3403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:44.764903 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:44.764907 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:44.765328 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:44.765711 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:44.765716 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:44.765719 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:44.765925 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:44.766449 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:03:44.766453 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:03:44.766641 (+ 188us) client_negotiation.cc:769] Sending connection context
0812 07:03:44.766782 (+ 141us) client_negotiation.cc:240] Negotiation successful
0812 07:03:44.767105 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14537,"thread_start_us":14362,"threads_started":1}
WARNING: no policy specified for kudu/127.6.61.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
I20260812 07:03:44.805545 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.2:0
--local_ip_for_outbound_sockets=127.6.61.2
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:39233
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:03:45.143056 8673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:03:45.143332 8673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:03:45.143501 8673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:03:45.150961 8673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:03:45.151098 8673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:03:45.151250 8673 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 07:03:45.151338 8673 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 07:03:45.151459 8673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.2
I20260812 07:03:45.161726 8673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.61.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.61.62:39233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.2
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:03:45.163038 8673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:03:45.164367 8673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:03:45.177595 8680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:45.177631 8681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:45.177556 8684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:03:45.178534 8673 server_base.cc:1034] running on GCE node
I20260812 07:03:45.179193 8673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:03:45.180074 8673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:03:45.181303 8673 hybrid_clock.cc:648] HybridClock initialized: now 1786518225181271 us; error 46 us; skew 500 ppm
I20260812 07:03:45.181615 8673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:03:45 dist-test-slave-69rg krb5kdc[8377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518225, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:03:45.184803 8673 init.cc:377] Logged in from keytab as kudu/127.6.61.2@KRBTEST.COM (short username kudu)
I20260812 07:03:45.186297 8673 webserver.cc:460] Webserver started at http://127.6.61.2:40193/ using document root <none> and password file <none>
I20260812 07:03:45.186988 8673 fs_manager.cc:359] Metadata directory not provided
I20260812 07:03:45.187137 8673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:03:45.187453 8673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:03:45.190380 8673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-1/data/instance:
uuid: "b873e04286174b76b4af16a01aea7cbb"
format_stamp: "Formatted at 2026-08-12 07:03:45 on dist-test-slave-69rg"
server_key: "1b05c1be7013c193b249d04ea5f1b943"
server_key_iv: "b80033a258f1b3c22cf552a88d037fdb"
server_key_version: "encryptionkey@0"
I20260812 07:03:45.191157 8673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance:
uuid: "b873e04286174b76b4af16a01aea7cbb"
format_stamp: "Formatted at 2026-08-12 07:03:45 on dist-test-slave-69rg"
server_key: "1b05c1be7013c193b249d04ea5f1b943"
server_key_iv: "b80033a258f1b3c22cf552a88d037fdb"
server_key_version: "encryptionkey@0"
I20260812 07:03:45.196182 8673 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 07:03:45.199514 8692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:45.200768 8673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 07:03:45.200943 8673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-1/wal
uuid: "b873e04286174b76b4af16a01aea7cbb"
format_stamp: "Formatted at 2026-08-12 07:03:45 on dist-test-slave-69rg"
server_key: "1b05c1be7013c193b249d04ea5f1b943"
server_key_iv: "b80033a258f1b3c22cf552a88d037fdb"
server_key_version: "encryptionkey@0"
I20260812 07:03:45.201138 8673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:03:45.230943 8673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:03:45.235244 8673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:03:45.235546 8673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:03:45.238596 8673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:03:45.238760 8673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:45.238895 8673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:03:45.238991 8673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 07:03:45 dist-test-slave-69rg krb5kdc[8377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518225, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:03:45.265792 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:45.239284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:43411 (local address 127.6.61.62:39233)
0812 07:03:45.239520 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:45.239527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:45.241183 (+ 1656us) server_negotiation.cc:407] Connection header received
0812 07:03:45.243756 (+ 2573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:45.243766 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:45.243879 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:45.243964 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:45.246868 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:45.247932 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:45.250510 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:45.251143 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:45.251306 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:45.257540 (+ 6234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:45.257550 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:45.257554 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:45.257604 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:45.261257 (+ 3653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:45.262578 (+ 1321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:45.262586 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:45.262593 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:45.262749 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:45.263442 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:45.263448 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:45.263452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:45.263966 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:45.264151 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:45.265172 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0812 07:03:45.265420 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260812 07:03:45.267969 8704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:45.239931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39233 (local address 127.6.61.2:43411)
0812 07:03:45.240962 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:45.241018 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:45.243331 (+ 2313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:45.244607 (+ 1276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:45.244632 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:45.245573 (+ 941us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:03:45.246638 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:45.246671 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:45.248170 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:45.248174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:45.250247 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:45.250262 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:45.251562 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:45.251574 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:45.251989 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:45.253117 (+ 1128us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:03:45.253156 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:03:45.257163 (+ 4007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:03:45.261442 (+ 4279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:45.261457 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:45.261487 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:45.262379 (+ 892us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:45.262899 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:45.262910 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:45.262916 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:45.263289 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:45.264130 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:03:45.264140 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:03:45.264617 (+ 477us) client_negotiation.cc:769] Sending connection context
0812 07:03:45.265031 (+ 414us) client_negotiation.cc:240] Negotiation successful
0812 07:03:45.265708 (+ 677us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":462,"threads_started":1}
I20260812 07:03:45.286113 8673 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.2:45959
I20260812 07:03:45.286262 8833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.2:45959 every 8 connection(s)
I20260812 07:03:45.287945 8673 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
I20260812 07:03:45.290611 8834 heartbeater.cc:347] Connected to a master server at 127.6.61.62:39233
I20260812 07:03:45.290868 8834 heartbeater.cc:464] Registering TS with master...
I20260812 07:03:45.291540 8834 heartbeater.cc:511] Master 127.6.61.62:39233 requested a full tablet report, sending...
I20260812 07:03:45.293072 8428 ts_manager.cc:194] Registered new tserver with Master: b873e04286174b76b4af16a01aea7cbb (127.6.61.2:45959)
I20260812 07:03:45.294350 8428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.2@KRBTEST.COM'} at 127.6.61.2:43411
I20260812 07:03:45.294579 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:45.276967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:38371 (local address 127.6.61.62:39233)
0812 07:03:45.277233 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:45.277243 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:45.278035 (+ 792us) server_negotiation.cc:407] Connection header received
0812 07:03:45.278348 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:45.278358 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:45.278470 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:45.278584 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:45.280371 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:45.281797 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:45.284301 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:45.284969 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:45.285130 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:45.287813 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:45.287818 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:45.287821 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:45.287856 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:45.291502 (+ 3646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:45.292295 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:45.292302 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:45.292306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:45.292489 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:45.292914 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:45.292917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:45.292919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:45.293356 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:45.293475 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:45.294007 (+ 532us) server_negotiation.cc:299] Negotiation successful
0812 07:03:45.294230 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 07:03:45.294668 8807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:45.276963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39233 (local address 127.6.61.2:38371)
0812 07:03:45.277900 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:45.277921 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:45.278163 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:45.278670 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:45.278683 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:45.279511 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:03:45.280073 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:45.280093 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:45.281977 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:45.281989 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:45.284115 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:45.284133 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:45.285319 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:45.285331 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:45.285580 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:45.286431 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:03:45.286454 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:03:45.287613 (+ 1159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:03:45.291736 (+ 4123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:45.291741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:45.291746 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:45.292168 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:45.292608 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:45.292611 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:45.292614 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:45.292797 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:45.293485 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:03:45.293491 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:03:45.293700 (+ 209us) client_negotiation.cc:769] Sending connection context
0812 07:03:45.293889 (+ 189us) client_negotiation.cc:240] Negotiation successful
0812 07:03:45.294224 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":514,"threads_started":1}
I20260812 07:03:45.297230 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 8673
I20260812 07:03:45.297457 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance
I20260812 07:03:45.298029 6388 external_mini_cluster.cc:1428] Setting key 312feb945a39ebb99863fa648fdb9369
WARNING: no policy specified for kudu/127.6.61.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
I20260812 07:03:45.347294 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.3:0
--local_ip_for_outbound_sockets=127.6.61.3
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:39233
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:03:45.680603 8838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:03:45.680939 8838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:03:45.681138 8838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:03:45.688455 8838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:03:45.688563 8838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:03:45.688760 8838 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 07:03:45.688844 8838 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 07:03:45.688982 8838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.3
I20260812 07:03:45.699021 8838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.61.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.61.62:39233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.3
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:03:45.700371 8838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:03:45.701702 8838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:03:45.715479 8846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:45.715518 8848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:45.715560 8850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:03:45.716643 8838 server_base.cc:1034] running on GCE node
I20260812 07:03:45.717352 8838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:03:45.718370 8838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:03:45.719573 8838 hybrid_clock.cc:648] HybridClock initialized: now 1786518225719527 us; error 38 us; skew 500 ppm
I20260812 07:03:45.719852 8838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:03:45 dist-test-slave-69rg krb5kdc[8377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518225, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:03:45.723253 8838 init.cc:377] Logged in from keytab as kudu/127.6.61.3@KRBTEST.COM (short username kudu)
I20260812 07:03:45.724854 8838 webserver.cc:460] Webserver started at http://127.6.61.3:46577/ using document root <none> and password file <none>
I20260812 07:03:45.725550 8838 fs_manager.cc:359] Metadata directory not provided
I20260812 07:03:45.725672 8838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:03:45.726058 8838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:03:45.729197 8838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-2/data/instance:
uuid: "529e37385608459bb7d952862fa9e191"
format_stamp: "Formatted at 2026-08-12 07:03:45 on dist-test-slave-69rg"
server_key: "d965f3ffff2bf1b6decf4efaba8ffd0c"
server_key_iv: "c460a33c5a416ec45aeb7decc3a0b1a0"
server_key_version: "encryptionkey@0"
I20260812 07:03:45.730072 8838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance:
uuid: "529e37385608459bb7d952862fa9e191"
format_stamp: "Formatted at 2026-08-12 07:03:45 on dist-test-slave-69rg"
server_key: "d965f3ffff2bf1b6decf4efaba8ffd0c"
server_key_iv: "c460a33c5a416ec45aeb7decc3a0b1a0"
server_key_version: "encryptionkey@0"
I20260812 07:03:45.735949 8838 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 07:03:45.739580 8857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:45.740942 8838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 07:03:45.741123 8838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-2/wal
uuid: "529e37385608459bb7d952862fa9e191"
format_stamp: "Formatted at 2026-08-12 07:03:45 on dist-test-slave-69rg"
server_key: "d965f3ffff2bf1b6decf4efaba8ffd0c"
server_key_iv: "c460a33c5a416ec45aeb7decc3a0b1a0"
server_key_version: "encryptionkey@0"
I20260812 07:03:45.741336 8838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:03:45.758471 8838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:03:45.763002 8838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:03:45.763329 8838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:03:45.766322 8838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:03:45.766424 8838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:45.766602 8838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:03:45.766673 8838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:45.768229 8669 heartbeater.cc:503] Master 127.6.61.62:39233 was elected leader, sending a full tablet report...
Aug 12 07:03:45 dist-test-slave-69rg krb5kdc[8377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518225, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:03:45.792754 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:45.767177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:40935 (local address 127.6.61.62:39233)
0812 07:03:45.767358 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:45.767363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:45.768752 (+ 1389us) server_negotiation.cc:407] Connection header received
0812 07:03:45.771479 (+ 2727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:45.771485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:45.771587 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:45.771737 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:45.774707 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:45.775960 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:45.778828 (+ 2868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:45.779414 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:45.779581 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:45.785717 (+ 6136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:45.785726 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:45.785733 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:45.785781 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:45.789347 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:45.790226 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:45.790239 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:45.790241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:45.790371 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:45.790879 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:45.790883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:45.790885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:45.791186 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:45.791348 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:45.792153 (+ 805us) server_negotiation.cc:299] Negotiation successful
0812 07:03:45.792386 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 07:03:45.793797 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:45.767629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39233 (local address 127.6.61.3:40935)
0812 07:03:45.768571 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:45.768618 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:45.771161 (+ 2543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:45.772248 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:45.772265 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:45.773258 (+ 993us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:03:45.774403 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:45.774440 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:45.776175 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:45.776179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:45.778629 (+ 2450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:45.778643 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:45.779751 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:45.779762 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:45.780050 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:45.781055 (+ 1005us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:03:45.781091 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:03:45.785353 (+ 4262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:03:45.789507 (+ 4154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:45.789517 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:45.789535 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:45.790079 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:45.790505 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:45.790509 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:45.790515 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:45.790737 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:45.791325 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:03:45.791333 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:03:45.791721 (+ 388us) client_negotiation.cc:769] Sending connection context
0812 07:03:45.791991 (+ 270us) client_negotiation.cc:240] Negotiation successful
0812 07:03:45.792545 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":1303,"threads_started":1}
I20260812 07:03:45.820355 8838 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.3:38361
I20260812 07:03:45.820456 8993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.3:38361 every 8 connection(s)
I20260812 07:03:45.822060 8838 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
I20260812 07:03:45.824620 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:45.805355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:54769 (local address 127.6.61.62:39233)
0812 07:03:45.805557 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:45.805563 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:45.806575 (+ 1012us) server_negotiation.cc:407] Connection header received
0812 07:03:45.806936 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:45.806944 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:45.807046 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:45.807132 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:45.809217 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:45.810488 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:45.813194 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:45.814020 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:45.814221 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:45.817667 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:45.817677 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:45.817684 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:45.817737 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:45.821168 (+ 3431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:45.822131 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:45.822136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:45.822138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:45.822348 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:45.822899 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:45.822904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:45.822907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:45.823275 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:45.823433 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:45.824130 (+ 697us) server_negotiation.cc:299] Negotiation successful
0812 07:03:45.824320 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 07:03:45.824618 8994 heartbeater.cc:347] Connected to a master server at 127.6.61.62:39233
I20260812 07:03:45.824939 8951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:45.805499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39233 (local address 127.6.61.3:54769)
0812 07:03:45.806447 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:45.806470 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:45.806740 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:45.807412 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:45.807418 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:45.808230 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:03:45.808991 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:45.809009 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:45.810656 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:45.810666 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:45.812995 (+ 2329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:45.813017 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:45.814330 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:45.814342 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:45.814654 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:45.815664 (+ 1010us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:03:45.815705 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:03:45.817431 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:03:45.821310 (+ 3879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:45.821315 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:45.821319 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:45.821971 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:45.822496 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:45.822502 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:45.822507 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:45.822745 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:45.823451 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:03:45.823460 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:03:45.823772 (+ 312us) client_negotiation.cc:769] Sending connection context
0812 07:03:45.823969 (+ 197us) client_negotiation.cc:240] Negotiation successful
0812 07:03:45.824333 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"thread_start_us":582,"threads_started":1}
I20260812 07:03:45.825402 8994 heartbeater.cc:464] Registering TS with master...
I20260812 07:03:45.826311 8994 heartbeater.cc:511] Master 127.6.61.62:39233 requested a full tablet report, sending...
I20260812 07:03:45.827888 8428 ts_manager.cc:194] Registered new tserver with Master: 529e37385608459bb7d952862fa9e191 (127.6.61.3:38361)
I20260812 07:03:45.829031 8428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.3@KRBTEST.COM'} at 127.6.61.3:40935
I20260812 07:03:45.829504 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 8838
I20260812 07:03:45.829699 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance
I20260812 07:03:45.830192 6388 external_mini_cluster.cc:1428] Setting key f34fd9d5d501db9cf4e564d090a5d726
I20260812 07:03:45.833134 6388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 07:03:45 dist-test-slave-69rg krb5kdc[8377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 07:03:45 dist-test-slave-69rg krb5kdc[8377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:03:45.888427 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:45.873767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54320 (local address 127.6.61.62:39233)
0812 07:03:45.874034 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:45.874041 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:45.874254 (+ 213us) server_negotiation.cc:407] Connection header received
0812 07:03:45.874494 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:45.874499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:45.874579 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:45.874692 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:45.876136 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:45.876879 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:45.878237 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:45.878687 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:45.878829 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:45.883020 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:45.883025 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:45.883028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:45.883065 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:45.885600 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:45.886331 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:45.886336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:45.886338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:45.886460 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:45.886886 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:45.886890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:45.886892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:45.887176 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:45.887439 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:45.887782 (+ 343us) server_negotiation.cc:299] Negotiation successful
0812 07:03:45.888029 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260812 07:03:45.893024 8428 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:03:45.900571 8428 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:03:45.933228 9015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:45.913058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:39565 (local address 127.0.0.1:37468)
0812 07:03:45.922701 (+ 9643us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:45.922750 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:45.922929 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:45.925418 (+ 2489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:45.925426 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:45.925454 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:03:45.925952 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:45.925968 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:45.927886 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:45.927895 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:45.930213 (+ 2318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:45.930229 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:45.932157 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:45.932164 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:45.932370 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:45.932423 (+ 53us) client_negotiation.cc:769] Sending connection context
0812 07:03:45.932550 (+ 127us) client_negotiation.cc:240] Negotiation successful
0812 07:03:45.932777 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9389,"mutex_wait_us":22,"thread_start_us":10834,"threads_started":1}
I20260812 07:03:45.933462 9011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:45.913295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37468 (local address 127.6.61.1:39565)
0812 07:03:45.924811 (+ 11516us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:45.924820 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:45.924852 (+ 32us) server_negotiation.cc:407] Connection header received
0812 07:03:45.924966 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:45.924974 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:45.925199 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:45.925411 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:03:45.926169 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:45.927706 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:45.930387 (+ 2681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:45.931960 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:45.932100 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:45.932321 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:45.932952 (+ 631us) server_negotiation.cc:299] Negotiation successful
0812 07:03:45.933143 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11351,"thread_start_us":11206,"threads_started":1}
I20260812 07:03:45.933714 9012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:45.913431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:45959 (local address 127.0.0.1:46138)
0812 07:03:45.923444 (+ 10013us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:45.923466 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:45.923610 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:45.926617 (+ 3007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:45.926621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:45.926649 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:03:45.927051 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:45.927065 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:45.928838 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:45.928845 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:45.930694 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:45.930710 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:45.932557 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:45.932564 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:45.932772 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:45.932968 (+ 196us) client_negotiation.cc:769] Sending connection context
0812 07:03:45.933053 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 07:03:45.933188 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9825,"mutex_wait_us":127,"thread_start_us":9662,"threads_started":1}
I20260812 07:03:45.934192 9016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:45.914513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43640 (local address 127.6.61.3:38361)
0812 07:03:45.925366 (+ 10853us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:45.925379 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:45.925437 (+ 58us) server_negotiation.cc:407] Connection header received
0812 07:03:45.925560 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:45.925570 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:45.925860 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:45.926092 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:03:45.926756 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:45.928502 (+ 1746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:45.931163 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:45.932399 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:45.933383 (+ 984us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:45.933607 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:45.933684 (+ 77us) server_negotiation.cc:299] Negotiation successful
0812 07:03:45.933871 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10635,"thread_start_us":10454,"threads_started":1}
I20260812 07:03:45.934613 9010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:45.913790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:38361 (local address 127.0.0.1:43640)
0812 07:03:45.924155 (+ 10365us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:45.924172 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:45.924291 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:45.926078 (+ 1787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:45.926083 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:45.926109 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:03:45.926494 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:45.926505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:45.928651 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:45.928656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:45.930954 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:45.930968 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:45.932653 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:45.932659 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:45.932856 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:45.932975 (+ 119us) client_negotiation.cc:769] Sending connection context
0812 07:03:45.933053 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 07:03:45.933219 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10211,"mutex_wait_us":51,"thread_start_us":8198,"threads_started":1}
I20260812 07:03:45.935638 9014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:45.913965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46138 (local address 127.6.61.2:45959)
0812 07:03:45.926054 (+ 12089us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:45.926065 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:45.926090 (+ 25us) server_negotiation.cc:407] Connection header received
0812 07:03:45.926192 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:45.926198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:45.926417 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:45.926614 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:03:45.927275 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:45.928687 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:45.930907 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:45.932282 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:45.932493 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:45.932704 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:45.935168 (+ 2464us) server_negotiation.cc:299] Negotiation successful
0812 07:03:45.935351 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11910,"thread_start_us":11762,"threads_started":1}
I20260812 07:03:45.937670 8586 tablet_service.cc:1467] Processing CreateTablet for tablet 221627d0921b4a808d037c7458a65191 (DEFAULT_TABLE table=test-table [id=7d5e460d3f1d4a488f0547a9cf34b9ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:03:45.938126 8913 tablet_service.cc:1467] Processing CreateTablet for tablet 221627d0921b4a808d037c7458a65191 (DEFAULT_TABLE table=test-table [id=7d5e460d3f1d4a488f0547a9cf34b9ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:03:45.939074 8586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 221627d0921b4a808d037c7458a65191. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:03:45.939476 8913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 221627d0921b4a808d037c7458a65191. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:03:45.939098 8752 tablet_service.cc:1467] Processing CreateTablet for tablet 221627d0921b4a808d037c7458a65191 (DEFAULT_TABLE table=test-table [id=7d5e460d3f1d4a488f0547a9cf34b9ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:03:45.940747 8752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 221627d0921b4a808d037c7458a65191. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:03:45.951402 9017 tablet_bootstrap.cc:492] T 221627d0921b4a808d037c7458a65191 P a6e3ffc000c840d1b0f8d22e592b556b: Bootstrap starting.
I20260812 07:03:45.954597 9018 tablet_bootstrap.cc:492] T 221627d0921b4a808d037c7458a65191 P 529e37385608459bb7d952862fa9e191: Bootstrap starting.
I20260812 07:03:45.955266 9017 tablet_bootstrap.cc:654] T 221627d0921b4a808d037c7458a65191 P a6e3ffc000c840d1b0f8d22e592b556b: Neither blocks nor log segments found. Creating new log.
I20260812 07:03:45.956350 9019 tablet_bootstrap.cc:492] T 221627d0921b4a808d037c7458a65191 P b873e04286174b76b4af16a01aea7cbb: Bootstrap starting.
I20260812 07:03:45.956643 9017 log.cc:824] T 221627d0921b4a808d037c7458a65191 P a6e3ffc000c840d1b0f8d22e592b556b: Log is configured to *not* fsync() on all Append() calls
I20260812 07:03:45.958137 9018 tablet_bootstrap.cc:654] T 221627d0921b4a808d037c7458a65191 P 529e37385608459bb7d952862fa9e191: Neither blocks nor log segments found. Creating new log.
I20260812 07:03:45.959453 9018 log.cc:824] T 221627d0921b4a808d037c7458a65191 P 529e37385608459bb7d952862fa9e191: Log is configured to *not* fsync() on all Append() calls
I20260812 07:03:45.959621 9019 tablet_bootstrap.cc:654] T 221627d0921b4a808d037c7458a65191 P b873e04286174b76b4af16a01aea7cbb: Neither blocks nor log segments found. Creating new log.
I20260812 07:03:45.960141 9017 tablet_bootstrap.cc:492] T 221627d0921b4a808d037c7458a65191 P a6e3ffc000c840d1b0f8d22e592b556b: No bootstrap required, opened a new log
I20260812 07:03:45.960506 9017 ts_tablet_manager.cc:1397] T 221627d0921b4a808d037c7458a65191 P a6e3ffc000c840d1b0f8d22e592b556b: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260812 07:03:45.961023 9019 log.cc:824] T 221627d0921b4a808d037c7458a65191 P b873e04286174b76b4af16a01aea7cbb: Log is configured to *not* fsync() on all Append() calls
I20260812 07:03:45.962719 9018 tablet_bootstrap.cc:492] T 221627d0921b4a808d037c7458a65191 P 529e37385608459bb7d952862fa9e191: No bootstrap required, opened a new log
I20260812 07:03:45.963079 9018 ts_tablet_manager.cc:1397] T 221627d0921b4a808d037c7458a65191 P 529e37385608459bb7d952862fa9e191: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260812 07:03:45.963850 9019 tablet_bootstrap.cc:492] T 221627d0921b4a808d037c7458a65191 P b873e04286174b76b4af16a01aea7cbb: No bootstrap required, opened a new log
I20260812 07:03:45.964097 9019 ts_tablet_manager.cc:1397] T 221627d0921b4a808d037c7458a65191 P b873e04286174b76b4af16a01aea7cbb: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260812 07:03:45.971175 9019 raft_consensus.cc:348] T 221627d0921b4a808d037c7458a65191 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:45.971606 9019 raft_consensus.cc:374] T 221627d0921b4a808d037c7458a65191 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:03:45.971714 9019 raft_consensus.cc:729] T 221627d0921b4a808d037c7458a65191 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b873e04286174b76b4af16a01aea7cbb, State: Initialized, Role: FOLLOWER
I20260812 07:03:45.971554 9017 raft_consensus.cc:348] T 221627d0921b4a808d037c7458a65191 P a6e3ffc000c840d1b0f8d22e592b556b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:45.972059 9017 raft_consensus.cc:374] T 221627d0921b4a808d037c7458a65191 P a6e3ffc000c840d1b0f8d22e592b556b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:03:45.972177 9017 raft_consensus.cc:729] T 221627d0921b4a808d037c7458a65191 P a6e3ffc000c840d1b0f8d22e592b556b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e3ffc000c840d1b0f8d22e592b556b, State: Initialized, Role: FOLLOWER
I20260812 07:03:45.972510 9019 consensus_queue.cc:260] T 221627d0921b4a808d037c7458a65191 P b873e04286174b76b4af16a01aea7cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:45.973143 9017 consensus_queue.cc:260] T 221627d0921b4a808d037c7458a65191 P a6e3ffc000c840d1b0f8d22e592b556b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:45.973888 9018 raft_consensus.cc:348] T 221627d0921b4a808d037c7458a65191 P 529e37385608459bb7d952862fa9e191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:45.974419 9018 raft_consensus.cc:374] T 221627d0921b4a808d037c7458a65191 P 529e37385608459bb7d952862fa9e191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:03:45.974557 9018 raft_consensus.cc:729] T 221627d0921b4a808d037c7458a65191 P 529e37385608459bb7d952862fa9e191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 529e37385608459bb7d952862fa9e191, State: Initialized, Role: FOLLOWER
I20260812 07:03:45.975296 9018 consensus_queue.cc:260] T 221627d0921b4a808d037c7458a65191 P 529e37385608459bb7d952862fa9e191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:45.976701 8994 heartbeater.cc:503] Master 127.6.61.62:39233 was elected leader, sending a full tablet report...
I20260812 07:03:45.977524 9018 ts_tablet_manager.cc:1428] T 221627d0921b4a808d037c7458a65191 P 529e37385608459bb7d952862fa9e191: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260812 07:03:45.979679 8834 heartbeater.cc:503] Master 127.6.61.62:39233 was elected leader, sending a full tablet report...
I20260812 07:03:45.980585 9019 ts_tablet_manager.cc:1428] T 221627d0921b4a808d037c7458a65191 P b873e04286174b76b4af16a01aea7cbb: Time spent starting tablet: real 0.016s user 0.005s sys 0.011s
I20260812 07:03:45.981074 9017 ts_tablet_manager.cc:1428] T 221627d0921b4a808d037c7458a65191 P a6e3ffc000c840d1b0f8d22e592b556b: Time spent starting tablet: real 0.020s user 0.012s sys 0.006s
W20260812 07:03:46.005455 8670 tablet.cc:2369] T 221627d0921b4a808d037c7458a65191 P a6e3ffc000c840d1b0f8d22e592b556b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 07:03:46.040962 8835 tablet.cc:2369] T 221627d0921b4a808d037c7458a65191 P b873e04286174b76b4af16a01aea7cbb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 07:03:46.074491 8995 tablet.cc:2369] T 221627d0921b4a808d037c7458a65191 P 529e37385608459bb7d952862fa9e191: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:03:46.171332 9023 raft_consensus.cc:482] T 221627d0921b4a808d037c7458a65191 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:03:46.171752 9023 raft_consensus.cc:504] T 221627d0921b4a808d037c7458a65191 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:46.173592 9023 leader_election.cc:288] T 221627d0921b4a808d037c7458a65191 P b873e04286174b76b4af16a01aea7cbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e3ffc000c840d1b0f8d22e592b556b (127.6.61.1:39565), 529e37385608459bb7d952862fa9e191 (127.6.61.3:38361)
I20260812 07:03:46.181273 9016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:46.174195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:41289 (local address 127.6.61.3:38361)
0812 07:03:46.174444 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:46.174450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:46.175630 (+ 1180us) server_negotiation.cc:407] Connection header received
0812 07:03:46.175849 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:46.175853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:46.175911 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:46.176013 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:03:46.176620 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.177337 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.179321 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.180139 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.180321 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:46.180447 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:46.180823 (+ 376us) server_negotiation.cc:299] Negotiation successful
0812 07:03:46.180977 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260812 07:03:46.181538 9026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:46.174036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:39565 (local address 127.6.61.2:43887)
0812 07:03:46.175131 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:46.175151 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:46.175351 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:46.175727 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:46.175732 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:46.175757 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:03:46.176133 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:46.176143 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.177150 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.177157 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:46.178901 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:46.178918 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.180390 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.180398 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:46.180633 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:46.180671 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 07:03:46.180786 (+ 115us) client_negotiation.cc:240] Negotiation successful
0812 07:03:46.181013 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":980,"mutex_wait_us":120,"thread_start_us":736,"threads_started":1}
I20260812 07:03:46.181617 9027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:46.174565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:38361 (local address 127.6.61.2:41289)
0812 07:03:46.175526 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:46.175540 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:46.175668 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:46.176035 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:46.176039 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:46.176069 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:03:46.176459 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:46.176467 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.177498 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.177508 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:46.179149 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:46.179162 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.180448 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.180455 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:46.180646 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:46.180687 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 07:03:46.180795 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 07:03:46.181019 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"mutex_wait_us":118,"thread_start_us":652,"threads_started":1}
I20260812 07:03:46.181315 9011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:46.174547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:43887 (local address 127.6.61.1:39565)
0812 07:03:46.174841 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:46.174848 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:46.175250 (+ 402us) server_negotiation.cc:407] Connection header received
0812 07:03:46.175517 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:46.175522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:46.175615 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:46.175712 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:03:46.176291 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.177019 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.179065 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.180090 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.180293 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:46.180426 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:46.180817 (+ 391us) server_negotiation.cc:299] Negotiation successful
0812 07:03:46.180974 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260812 07:03:46.182750 8936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221627d0921b4a808d037c7458a65191" candidate_uuid: "b873e04286174b76b4af16a01aea7cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529e37385608459bb7d952862fa9e191" is_pre_election: true
I20260812 07:03:46.182819 8611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221627d0921b4a808d037c7458a65191" candidate_uuid: "b873e04286174b76b4af16a01aea7cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" is_pre_election: true
I20260812 07:03:46.183281 8936 raft_consensus.cc:2454] T 221627d0921b4a808d037c7458a65191 P 529e37385608459bb7d952862fa9e191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b873e04286174b76b4af16a01aea7cbb in term 0.
I20260812 07:03:46.183317 8611 raft_consensus.cc:2454] T 221627d0921b4a808d037c7458a65191 P a6e3ffc000c840d1b0f8d22e592b556b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b873e04286174b76b4af16a01aea7cbb in term 0.
I20260812 07:03:46.183977 8693 leader_election.cc:302] T 221627d0921b4a808d037c7458a65191 P b873e04286174b76b4af16a01aea7cbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e3ffc000c840d1b0f8d22e592b556b, b873e04286174b76b4af16a01aea7cbb; no voters:
I20260812 07:03:46.184377 9023 raft_consensus.cc:2781] T 221627d0921b4a808d037c7458a65191 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:03:46.184508 9023 raft_consensus.cc:482] T 221627d0921b4a808d037c7458a65191 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:03:46.184628 9023 raft_consensus.cc:3037] T 221627d0921b4a808d037c7458a65191 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:03:46.187022 9023 raft_consensus.cc:504] T 221627d0921b4a808d037c7458a65191 P b873e04286174b76b4af16a01aea7cbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:46.187711 9023 leader_election.cc:288] T 221627d0921b4a808d037c7458a65191 P b873e04286174b76b4af16a01aea7cbb [CANDIDATE]: Term 1 election: Requested vote from peers a6e3ffc000c840d1b0f8d22e592b556b (127.6.61.1:39565), 529e37385608459bb7d952862fa9e191 (127.6.61.3:38361)
I20260812 07:03:46.188192 8936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221627d0921b4a808d037c7458a65191" candidate_uuid: "b873e04286174b76b4af16a01aea7cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529e37385608459bb7d952862fa9e191"
I20260812 07:03:46.188447 8936 raft_consensus.cc:3037] T 221627d0921b4a808d037c7458a65191 P 529e37385608459bb7d952862fa9e191 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:03:46.188499 8611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221627d0921b4a808d037c7458a65191" candidate_uuid: "b873e04286174b76b4af16a01aea7cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e3ffc000c840d1b0f8d22e592b556b"
I20260812 07:03:46.188735 8611 raft_consensus.cc:3037] T 221627d0921b4a808d037c7458a65191 P a6e3ffc000c840d1b0f8d22e592b556b [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:03:46.190939 8936 raft_consensus.cc:2454] T 221627d0921b4a808d037c7458a65191 P 529e37385608459bb7d952862fa9e191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b873e04286174b76b4af16a01aea7cbb in term 1.
I20260812 07:03:46.191133 8611 raft_consensus.cc:2454] T 221627d0921b4a808d037c7458a65191 P a6e3ffc000c840d1b0f8d22e592b556b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b873e04286174b76b4af16a01aea7cbb in term 1.
I20260812 07:03:46.191423 8696 leader_election.cc:302] T 221627d0921b4a808d037c7458a65191 P b873e04286174b76b4af16a01aea7cbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 529e37385608459bb7d952862fa9e191, b873e04286174b76b4af16a01aea7cbb; no voters:
I20260812 07:03:46.191736 9023 raft_consensus.cc:2781] T 221627d0921b4a808d037c7458a65191 P b873e04286174b76b4af16a01aea7cbb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:03:46.192235 9023 raft_consensus.cc:686] T 221627d0921b4a808d037c7458a65191 P b873e04286174b76b4af16a01aea7cbb [term 1 LEADER]: Becoming Leader. State: Replica: b873e04286174b76b4af16a01aea7cbb, State: Running, Role: LEADER
I20260812 07:03:46.192721 9023 consensus_queue.cc:237] T 221627d0921b4a808d037c7458a65191 P b873e04286174b76b4af16a01aea7cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:46.197865 8428 catalog_manager.cc:5495] T 221627d0921b4a808d037c7458a65191 P b873e04286174b76b4af16a01aea7cbb reported cstate change: term changed from 0 to 1, leader changed from <none> to b873e04286174b76b4af16a01aea7cbb (127.6.61.2). New cstate: current_term: 1 leader_uuid: "b873e04286174b76b4af16a01aea7cbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } health_report { overall_health: UNKNOWN } } }
Aug 12 07:03:46 dist-test-slave-69rg krb5kdc[8377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518224, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:03:46.278255 9035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:46.264480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39233 (local address 127.0.0.1:54334)
0812 07:03:46.265195 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:46.265219 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:46.265361 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:46.265689 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:46.265692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:46.266497 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:03:46.266780 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:46.266790 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.267793 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.267797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:46.268792 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:46.268803 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.269536 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.269540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:46.269664 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:46.270422 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:03:46.270439 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:03:46.272769 (+ 2330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:03:46.275599 (+ 2830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:46.275614 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:46.275623 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:46.276074 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:46.276505 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:46.276509 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:46.276512 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:46.276681 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:46.277200 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:03:46.277207 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:03:46.277370 (+ 163us) client_negotiation.cc:769] Sending connection context
0812 07:03:46.277539 (+ 169us) client_negotiation.cc:240] Negotiation successful
0812 07:03:46.277788 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":404,"threads_started":1}
I20260812 07:03:46.278258 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:46.264457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54334 (local address 127.6.61.62:39233)
0812 07:03:46.264708 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:46.264716 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:46.265316 (+ 600us) server_negotiation.cc:407] Connection header received
0812 07:03:46.265475 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:46.265479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:46.265545 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:46.265630 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:46.266920 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.267656 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.268926 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.269365 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.269459 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:46.272943 (+ 3484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:46.272948 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:46.272951 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:46.272979 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:46.275446 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:46.276234 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:46.276238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:46.276240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:46.276370 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:46.276799 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:46.276803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:46.276805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:46.277051 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:46.277158 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:46.277614 (+ 456us) server_negotiation.cc:299] Negotiation successful
0812 07:03:46.278007 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":122,"server-negotiator.queue_time_us":74}
I20260812 07:03:46.283496 8428 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:54334:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 07:03:46.284598 8428 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:03:46.291488 8586 tablet_service.cc:1467] Processing CreateTablet for tablet 00ad7ce8de6b480a8c17e3302eb0b9a9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a0f57e78b9e4478b9f92bf7fa8303773]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 07:03:46.291564 8752 tablet_service.cc:1467] Processing CreateTablet for tablet 00ad7ce8de6b480a8c17e3302eb0b9a9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a0f57e78b9e4478b9f92bf7fa8303773]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 07:03:46.292435 8752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00ad7ce8de6b480a8c17e3302eb0b9a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:03:46.292434 8586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00ad7ce8de6b480a8c17e3302eb0b9a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:03:46.295804 8913 tablet_service.cc:1467] Processing CreateTablet for tablet 00ad7ce8de6b480a8c17e3302eb0b9a9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a0f57e78b9e4478b9f92bf7fa8303773]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 07:03:46.296833 8913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00ad7ce8de6b480a8c17e3302eb0b9a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:03:46.300194 9017 tablet_bootstrap.cc:492] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P a6e3ffc000c840d1b0f8d22e592b556b: Bootstrap starting.
I20260812 07:03:46.300885 9019 tablet_bootstrap.cc:492] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P b873e04286174b76b4af16a01aea7cbb: Bootstrap starting.
I20260812 07:03:46.302042 9017 tablet_bootstrap.cc:654] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P a6e3ffc000c840d1b0f8d22e592b556b: Neither blocks nor log segments found. Creating new log.
I20260812 07:03:46.302706 9019 tablet_bootstrap.cc:654] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P b873e04286174b76b4af16a01aea7cbb: Neither blocks nor log segments found. Creating new log.
I20260812 07:03:46.304836 9017 tablet_bootstrap.cc:492] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P a6e3ffc000c840d1b0f8d22e592b556b: No bootstrap required, opened a new log
I20260812 07:03:46.305002 9017 ts_tablet_manager.cc:1397] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P a6e3ffc000c840d1b0f8d22e592b556b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260812 07:03:46.305553 9019 tablet_bootstrap.cc:492] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P b873e04286174b76b4af16a01aea7cbb: No bootstrap required, opened a new log
I20260812 07:03:46.305721 9019 ts_tablet_manager.cc:1397] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P b873e04286174b76b4af16a01aea7cbb: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260812 07:03:46.305907 9017 raft_consensus.cc:348] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P a6e3ffc000c840d1b0f8d22e592b556b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:46.306208 9017 raft_consensus.cc:374] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P a6e3ffc000c840d1b0f8d22e592b556b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:03:46.306324 9017 raft_consensus.cc:729] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P a6e3ffc000c840d1b0f8d22e592b556b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e3ffc000c840d1b0f8d22e592b556b, State: Initialized, Role: FOLLOWER
I20260812 07:03:46.306586 9017 consensus_queue.cc:260] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P a6e3ffc000c840d1b0f8d22e592b556b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:46.306634 9019 raft_consensus.cc:348] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:46.306849 9019 raft_consensus.cc:374] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:03:46.306950 9019 raft_consensus.cc:729] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b873e04286174b76b4af16a01aea7cbb, State: Initialized, Role: FOLLOWER
I20260812 07:03:46.307219 9019 consensus_queue.cc:260] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P b873e04286174b76b4af16a01aea7cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:46.307102 9024 tablet_replica.cc:439] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P a6e3ffc000c840d1b0f8d22e592b556b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } } }
I20260812 07:03:46.307423 9018 tablet_bootstrap.cc:492] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P 529e37385608459bb7d952862fa9e191: Bootstrap starting.
I20260812 07:03:46.307511 9024 tablet_replica.cc:442] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P a6e3ffc000c840d1b0f8d22e592b556b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:03:46.307646 9023 tablet_replica.cc:439] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P b873e04286174b76b4af16a01aea7cbb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } } }
I20260812 07:03:46.307906 9023 tablet_replica.cc:442] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P b873e04286174b76b4af16a01aea7cbb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:03:46.308156 9017 ts_tablet_manager.cc:1428] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P a6e3ffc000c840d1b0f8d22e592b556b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260812 07:03:46.308897 9019 ts_tablet_manager.cc:1428] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P b873e04286174b76b4af16a01aea7cbb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 07:03:46.309329 9018 tablet_bootstrap.cc:654] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P 529e37385608459bb7d952862fa9e191: Neither blocks nor log segments found. Creating new log.
I20260812 07:03:46.312516 9018 tablet_bootstrap.cc:492] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P 529e37385608459bb7d952862fa9e191: No bootstrap required, opened a new log
I20260812 07:03:46.312846 9018 ts_tablet_manager.cc:1397] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P 529e37385608459bb7d952862fa9e191: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20260812 07:03:46.313537 9018 raft_consensus.cc:348] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P 529e37385608459bb7d952862fa9e191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:46.313758 9018 raft_consensus.cc:374] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P 529e37385608459bb7d952862fa9e191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:03:46.313911 9018 raft_consensus.cc:729] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P 529e37385608459bb7d952862fa9e191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 529e37385608459bb7d952862fa9e191, State: Initialized, Role: FOLLOWER
I20260812 07:03:46.314141 9018 consensus_queue.cc:260] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P 529e37385608459bb7d952862fa9e191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:46.314558 9025 tablet_replica.cc:439] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P 529e37385608459bb7d952862fa9e191: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } } }
I20260812 07:03:46.314752 9018 ts_tablet_manager.cc:1428] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P 529e37385608459bb7d952862fa9e191: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260812 07:03:46.314904 9025 tablet_replica.cc:442] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P 529e37385608459bb7d952862fa9e191: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:03:46.421116 9023 raft_consensus.cc:482] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:03:46.421459 9023 raft_consensus.cc:504] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:46.422326 9023 leader_election.cc:288] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P b873e04286174b76b4af16a01aea7cbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e3ffc000c840d1b0f8d22e592b556b (127.6.61.1:39565), 529e37385608459bb7d952862fa9e191 (127.6.61.3:38361)
I20260812 07:03:46.423127 8611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ad7ce8de6b480a8c17e3302eb0b9a9" candidate_uuid: "b873e04286174b76b4af16a01aea7cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" is_pre_election: true
I20260812 07:03:46.423132 8936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ad7ce8de6b480a8c17e3302eb0b9a9" candidate_uuid: "b873e04286174b76b4af16a01aea7cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529e37385608459bb7d952862fa9e191" is_pre_election: true
I20260812 07:03:46.423453 8611 raft_consensus.cc:2454] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P a6e3ffc000c840d1b0f8d22e592b556b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b873e04286174b76b4af16a01aea7cbb in term 0.
I20260812 07:03:46.423460 8936 raft_consensus.cc:2454] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P 529e37385608459bb7d952862fa9e191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b873e04286174b76b4af16a01aea7cbb in term 0.
I20260812 07:03:46.424033 8696 leader_election.cc:302] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P b873e04286174b76b4af16a01aea7cbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 529e37385608459bb7d952862fa9e191, a6e3ffc000c840d1b0f8d22e592b556b, b873e04286174b76b4af16a01aea7cbb; no voters:
I20260812 07:03:46.424494 9023 raft_consensus.cc:2781] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:03:46.424616 9023 raft_consensus.cc:482] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:03:46.424739 9023 raft_consensus.cc:3037] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:03:46.427381 9023 raft_consensus.cc:504] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P b873e04286174b76b4af16a01aea7cbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:46.428179 9023 leader_election.cc:288] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P b873e04286174b76b4af16a01aea7cbb [CANDIDATE]: Term 1 election: Requested vote from peers a6e3ffc000c840d1b0f8d22e592b556b (127.6.61.1:39565), 529e37385608459bb7d952862fa9e191 (127.6.61.3:38361)
I20260812 07:03:46.428887 8611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ad7ce8de6b480a8c17e3302eb0b9a9" candidate_uuid: "b873e04286174b76b4af16a01aea7cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e3ffc000c840d1b0f8d22e592b556b"
I20260812 07:03:46.428887 8936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ad7ce8de6b480a8c17e3302eb0b9a9" candidate_uuid: "b873e04286174b76b4af16a01aea7cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529e37385608459bb7d952862fa9e191"
I20260812 07:03:46.429198 8936 raft_consensus.cc:3037] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P 529e37385608459bb7d952862fa9e191 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:03:46.429196 8611 raft_consensus.cc:3037] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P a6e3ffc000c840d1b0f8d22e592b556b [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:03:46.432540 8611 raft_consensus.cc:2454] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P a6e3ffc000c840d1b0f8d22e592b556b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b873e04286174b76b4af16a01aea7cbb in term 1.
I20260812 07:03:46.432540 8936 raft_consensus.cc:2454] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P 529e37385608459bb7d952862fa9e191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b873e04286174b76b4af16a01aea7cbb in term 1.
I20260812 07:03:46.433220 8696 leader_election.cc:302] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P b873e04286174b76b4af16a01aea7cbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 529e37385608459bb7d952862fa9e191, b873e04286174b76b4af16a01aea7cbb; no voters:
I20260812 07:03:46.433586 9023 raft_consensus.cc:2781] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P b873e04286174b76b4af16a01aea7cbb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:03:46.433804 9023 raft_consensus.cc:686] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P b873e04286174b76b4af16a01aea7cbb [term 1 LEADER]: Becoming Leader. State: Replica: b873e04286174b76b4af16a01aea7cbb, State: Running, Role: LEADER
I20260812 07:03:46.434240 9023 consensus_queue.cc:237] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P b873e04286174b76b4af16a01aea7cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:46.436499 9028 tablet_replica.cc:439] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P b873e04286174b76b4af16a01aea7cbb: TxnStatusTablet state changed. Reason: New leader b873e04286174b76b4af16a01aea7cbb. Latest consensus state: current_term: 1 leader_uuid: "b873e04286174b76b4af16a01aea7cbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } } }
I20260812 07:03:46.436792 9028 tablet_replica.cc:442] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P b873e04286174b76b4af16a01aea7cbb: This TxnStatusTablet replica's current role is: LEADER
I20260812 07:03:46.437522 9040 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 07:03:46.437902 9040 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 07:03:46.438241 8427 catalog_manager.cc:5495] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P b873e04286174b76b4af16a01aea7cbb reported cstate change: term changed from 0 to 1, leader changed from <none> to b873e04286174b76b4af16a01aea7cbb (127.6.61.2). New cstate: current_term: 1 leader_uuid: "b873e04286174b76b4af16a01aea7cbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } health_report { overall_health: UNKNOWN } } }
I20260812 07:03:46.474716 9035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:46.466381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:45959 (local address 127.0.0.1:46140)
0812 07:03:46.466744 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:46.466769 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:46.466933 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:46.468173 (+ 1240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:46.468177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:46.468188 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 07:03:46.468383 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:46.468391 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.470226 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.470231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:46.471559 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:46.471573 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.472748 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.472754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:46.472890 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:46.472954 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 07:03:46.474254 (+ 1300us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 07:03:46.474277 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 07:03:46.474359 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 07:03:46.474489 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260812 07:03:46.474989 9043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:46.466588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46140 (local address 127.6.61.2:45959)
0812 07:03:46.467600 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:46.467611 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:46.467632 (+ 21us) server_negotiation.cc:407] Connection header received
0812 07:03:46.467782 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:46.467790 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:46.467914 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:46.468132 (+ 218us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:03:46.468558 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.470057 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.471771 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.472535 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.472706 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:46.473226 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:03:46.474134 (+ 908us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:03:46.474300 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:46.474417 (+ 117us) server_negotiation.cc:299] Negotiation successful
0812 07:03:46.474524 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":825,"thread_start_us":640,"threads_started":1}
I20260812 07:03:46.483197 8611 raft_consensus.cc:1261] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P a6e3ffc000c840d1b0f8d22e592b556b [term 1 FOLLOWER]: Refusing update from remote peer b873e04286174b76b4af16a01aea7cbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:03:46.483440 8936 raft_consensus.cc:1261] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P 529e37385608459bb7d952862fa9e191 [term 1 FOLLOWER]: Refusing update from remote peer b873e04286174b76b4af16a01aea7cbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:03:46.484335 9028 consensus_queue.cc:1035] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P b873e04286174b76b4af16a01aea7cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:03:46.484663 9023 consensus_queue.cc:1035] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P b873e04286174b76b4af16a01aea7cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:03:46.492748 9024 tablet_replica.cc:439] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P a6e3ffc000c840d1b0f8d22e592b556b: TxnStatusTablet state changed. Reason: New leader b873e04286174b76b4af16a01aea7cbb. Latest consensus state: current_term: 1 leader_uuid: "b873e04286174b76b4af16a01aea7cbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } } }
I20260812 07:03:46.493103 9024 tablet_replica.cc:442] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P a6e3ffc000c840d1b0f8d22e592b556b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:03:46.496078 9044 mvcc.cc:205] Tried to move back new op lower bound from 7317578655664615424 to 7317578655478091776. Current Snapshot: MvccSnapshot[applied={T|T < 7317578655664615424}]
I20260812 07:03:46.496865 9023 tablet_replica.cc:439] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P b873e04286174b76b4af16a01aea7cbb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b873e04286174b76b4af16a01aea7cbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } } }
I20260812 07:03:46.497218 9023 tablet_replica.cc:442] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P b873e04286174b76b4af16a01aea7cbb: This TxnStatusTablet replica's current role is: LEADER
I20260812 07:03:46.500541 9024 tablet_replica.cc:439] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P a6e3ffc000c840d1b0f8d22e592b556b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b873e04286174b76b4af16a01aea7cbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } } }
I20260812 07:03:46.500824 9024 tablet_replica.cc:442] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P a6e3ffc000c840d1b0f8d22e592b556b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:03:46.510425 9028 tablet_replica.cc:439] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P b873e04286174b76b4af16a01aea7cbb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b873e04286174b76b4af16a01aea7cbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } } }
I20260812 07:03:46.510733 9028 tablet_replica.cc:442] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P b873e04286174b76b4af16a01aea7cbb: This TxnStatusTablet replica's current role is: LEADER
I20260812 07:03:46.514115 9025 tablet_replica.cc:439] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P 529e37385608459bb7d952862fa9e191: TxnStatusTablet state changed. Reason: New leader b873e04286174b76b4af16a01aea7cbb. Latest consensus state: current_term: 1 leader_uuid: "b873e04286174b76b4af16a01aea7cbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } } }
I20260812 07:03:46.514519 9025 tablet_replica.cc:442] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P 529e37385608459bb7d952862fa9e191: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:03:46.518975 9025 tablet_replica.cc:439] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P 529e37385608459bb7d952862fa9e191: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b873e04286174b76b4af16a01aea7cbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } } }
I20260812 07:03:46.519243 9025 tablet_replica.cc:442] T 00ad7ce8de6b480a8c17e3302eb0b9a9 P 529e37385608459bb7d952862fa9e191: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:03:46.528852 9043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:46.522134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46148 (local address 127.6.61.2:45959)
0812 07:03:46.522313 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:46.522318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:46.522854 (+ 536us) server_negotiation.cc:407] Connection header received
0812 07:03:46.522998 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:46.523004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:46.523100 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:46.523253 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:03:46.523720 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.525102 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.526615 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.527333 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.527485 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:46.527916 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:03:46.528061 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:03:46.528172 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:46.528378 (+ 206us) server_negotiation.cc:299] Negotiation successful
0812 07:03:46.528475 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 07:03:46.549331 9058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:46.541724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:45959 (local address 127.6.61.2:45149)
0812 07:03:46.542561 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:46.542592 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:46.542748 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:46.543146 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:46.543149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:46.543159 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 07:03:46.543663 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:46.543680 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.545371 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.545376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:46.546887 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:46.546900 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.548007 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.548013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:46.548133 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:46.548210 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 07:03:46.548707 (+ 497us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 07:03:46.548716 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 07:03:46.548782 (+ 66us) client_negotiation.cc:240] Negotiation successful
0812 07:03:46.548890 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":505,"threads_started":1}
I20260812 07:03:46.549340 9043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:46.541925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:45149 (local address 127.6.61.2:45959)
0812 07:03:46.542180 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:46.542187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:46.542697 (+ 510us) server_negotiation.cc:407] Connection header received
0812 07:03:46.542889 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:46.542895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:46.543019 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:46.543131 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:03:46.543841 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.545235 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.547031 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.547781 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.547936 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:46.548458 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:03:46.548600 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:03:46.548709 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:46.548837 (+ 128us) server_negotiation.cc:299] Negotiation successful
0812 07:03:46.548931 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 07:03:46.558959 8936 raft_consensus.cc:1261] T 221627d0921b4a808d037c7458a65191 P 529e37385608459bb7d952862fa9e191 [term 1 FOLLOWER]: Refusing update from remote peer b873e04286174b76b4af16a01aea7cbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:03:46.559064 8611 raft_consensus.cc:1261] T 221627d0921b4a808d037c7458a65191 P a6e3ffc000c840d1b0f8d22e592b556b [term 1 FOLLOWER]: Refusing update from remote peer b873e04286174b76b4af16a01aea7cbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:03:46.559510 9028 consensus_queue.cc:1035] T 221627d0921b4a808d037c7458a65191 P b873e04286174b76b4af16a01aea7cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:03:46.559787 9023 consensus_queue.cc:1035] T 221627d0921b4a808d037c7458a65191 P b873e04286174b76b4af16a01aea7cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:03:46.566246 9045 mvcc.cc:205] Tried to move back new op lower bound from 7317578655980359680 to 7317578654489284608. Current Snapshot: MvccSnapshot[applied={T|T < 7317578655980359680}]
I20260812 07:03:46.623277 8427 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:03:46.623555 8427 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:03:46.627089 8427 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 100f02a2c2c24a9f9f301631ca439064: Sending DeleteTablet for 3 replicas of tablet 221627d0921b4a808d037c7458a65191
I20260812 07:03:46.628249 8586 tablet_service.cc:1514] Processing DeleteTablet for tablet 221627d0921b4a808d037c7458a65191 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:03:46 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:37468
I20260812 07:03:46.628640 8913 tablet_service.cc:1514] Processing DeleteTablet for tablet 221627d0921b4a808d037c7458a65191 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:03:46 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:43640
I20260812 07:03:46.628989 8752 tablet_service.cc:1514] Processing DeleteTablet for tablet 221627d0921b4a808d037c7458a65191 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:03:46 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:46138
I20260812 07:03:46.630347 8427 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:03:46.631428 8427 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:03:46.634353 9069 tablet_replica.cc:330] T 221627d0921b4a808d037c7458a65191 P b873e04286174b76b4af16a01aea7cbb: stopping tablet replica
I20260812 07:03:46.634907 9068 tablet_replica.cc:330] T 221627d0921b4a808d037c7458a65191 P 529e37385608459bb7d952862fa9e191: stopping tablet replica
I20260812 07:03:46.640435 8752 tablet_service.cc:1467] Processing CreateTablet for tablet 92eb7400aa9a4d5587aff85cc50eccc7 (DEFAULT_TABLE table=test-table [id=45d7c8ba52404d7793311350110ac8f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:03:46.640479 8585 tablet_service.cc:1467] Processing CreateTablet for tablet 92eb7400aa9a4d5587aff85cc50eccc7 (DEFAULT_TABLE table=test-table [id=45d7c8ba52404d7793311350110ac8f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:03:46.640971 8752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92eb7400aa9a4d5587aff85cc50eccc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:03:46.641049 8585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92eb7400aa9a4d5587aff85cc50eccc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:03:46.648307 9019 tablet_bootstrap.cc:492] T 92eb7400aa9a4d5587aff85cc50eccc7 P b873e04286174b76b4af16a01aea7cbb: Bootstrap starting.
I20260812 07:03:46.650163 9019 tablet_bootstrap.cc:654] T 92eb7400aa9a4d5587aff85cc50eccc7 P b873e04286174b76b4af16a01aea7cbb: Neither blocks nor log segments found. Creating new log.
I20260812 07:03:46.653960 9017 tablet_bootstrap.cc:492] T 92eb7400aa9a4d5587aff85cc50eccc7 P a6e3ffc000c840d1b0f8d22e592b556b: Bootstrap starting.
I20260812 07:03:46.656186 9017 tablet_bootstrap.cc:654] T 92eb7400aa9a4d5587aff85cc50eccc7 P a6e3ffc000c840d1b0f8d22e592b556b: Neither blocks nor log segments found. Creating new log.
I20260812 07:03:46.656505 9067 tablet_replica.cc:330] T 221627d0921b4a808d037c7458a65191 P a6e3ffc000c840d1b0f8d22e592b556b: stopping tablet replica
I20260812 07:03:46.657008 9067 raft_consensus.cc:2229] T 221627d0921b4a808d037c7458a65191 P a6e3ffc000c840d1b0f8d22e592b556b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:03:46.657477 9067 raft_consensus.cc:2258] T 221627d0921b4a808d037c7458a65191 P a6e3ffc000c840d1b0f8d22e592b556b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:03:46.659224 9017 tablet_bootstrap.cc:492] T 92eb7400aa9a4d5587aff85cc50eccc7 P a6e3ffc000c840d1b0f8d22e592b556b: No bootstrap required, opened a new log
I20260812 07:03:46.659278 9019 tablet_bootstrap.cc:492] T 92eb7400aa9a4d5587aff85cc50eccc7 P b873e04286174b76b4af16a01aea7cbb: No bootstrap required, opened a new log
I20260812 07:03:46.659467 9019 ts_tablet_manager.cc:1397] T 92eb7400aa9a4d5587aff85cc50eccc7 P b873e04286174b76b4af16a01aea7cbb: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260812 07:03:46.659596 9017 ts_tablet_manager.cc:1397] T 92eb7400aa9a4d5587aff85cc50eccc7 P a6e3ffc000c840d1b0f8d22e592b556b: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260812 07:03:46.660144 9067 ts_tablet_manager.cc:1905] T 221627d0921b4a808d037c7458a65191 P a6e3ffc000c840d1b0f8d22e592b556b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:03:46.660347 9019 raft_consensus.cc:348] T 92eb7400aa9a4d5587aff85cc50eccc7 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:46.660616 9019 raft_consensus.cc:374] T 92eb7400aa9a4d5587aff85cc50eccc7 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:03:46.660719 9019 raft_consensus.cc:729] T 92eb7400aa9a4d5587aff85cc50eccc7 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b873e04286174b76b4af16a01aea7cbb, State: Initialized, Role: FOLLOWER
I20260812 07:03:46.660725 9017 raft_consensus.cc:348] T 92eb7400aa9a4d5587aff85cc50eccc7 P a6e3ffc000c840d1b0f8d22e592b556b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:46.660997 9019 consensus_queue.cc:260] T 92eb7400aa9a4d5587aff85cc50eccc7 P b873e04286174b76b4af16a01aea7cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:46.661211 9017 raft_consensus.cc:374] T 92eb7400aa9a4d5587aff85cc50eccc7 P a6e3ffc000c840d1b0f8d22e592b556b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:03:46.661418 9017 raft_consensus.cc:729] T 92eb7400aa9a4d5587aff85cc50eccc7 P a6e3ffc000c840d1b0f8d22e592b556b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e3ffc000c840d1b0f8d22e592b556b, State: Initialized, Role: FOLLOWER
I20260812 07:03:46.662282 9019 ts_tablet_manager.cc:1428] T 92eb7400aa9a4d5587aff85cc50eccc7 P b873e04286174b76b4af16a01aea7cbb: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260812 07:03:46.662802 9017 consensus_queue.cc:260] T 92eb7400aa9a4d5587aff85cc50eccc7 P a6e3ffc000c840d1b0f8d22e592b556b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:46.663385 9017 ts_tablet_manager.cc:1428] T 92eb7400aa9a4d5587aff85cc50eccc7 P a6e3ffc000c840d1b0f8d22e592b556b: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260812 07:03:46.647382 8913 tablet_service.cc:1467] Processing CreateTablet for tablet 92eb7400aa9a4d5587aff85cc50eccc7 (DEFAULT_TABLE table=test-table [id=45d7c8ba52404d7793311350110ac8f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:03:46.666409 8913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92eb7400aa9a4d5587aff85cc50eccc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:03:46.669078 9067 ts_tablet_manager.cc:1918] T 221627d0921b4a808d037c7458a65191 P a6e3ffc000c840d1b0f8d22e592b556b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 07:03:46.669275 9067 log.cc:1196] T 221627d0921b4a808d037c7458a65191 P a6e3ffc000c840d1b0f8d22e592b556b: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-0/wal/wals/221627d0921b4a808d037c7458a65191
I20260812 07:03:46.669919 9067 ts_tablet_manager.cc:1939] T 221627d0921b4a808d037c7458a65191 P a6e3ffc000c840d1b0f8d22e592b556b: Deleting consensus metadata
I20260812 07:03:46.671893 8408 catalog_manager.cc:4841] TS a6e3ffc000c840d1b0f8d22e592b556b (127.6.61.1:39565): tablet 221627d0921b4a808d037c7458a65191 (table test-table [id=7d5e460d3f1d4a488f0547a9cf34b9ff]) successfully deleted
I20260812 07:03:46.673295 9018 tablet_bootstrap.cc:492] T 92eb7400aa9a4d5587aff85cc50eccc7 P 529e37385608459bb7d952862fa9e191: Bootstrap starting.
I20260812 07:03:46.673676 9069 raft_consensus.cc:2229] T 221627d0921b4a808d037c7458a65191 P b873e04286174b76b4af16a01aea7cbb [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:03:46.674401 9069 raft_consensus.cc:2258] T 221627d0921b4a808d037c7458a65191 P b873e04286174b76b4af16a01aea7cbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:03:46.674902 9018 tablet_bootstrap.cc:654] T 92eb7400aa9a4d5587aff85cc50eccc7 P 529e37385608459bb7d952862fa9e191: Neither blocks nor log segments found. Creating new log.
I20260812 07:03:46.675076 9068 raft_consensus.cc:2229] T 221627d0921b4a808d037c7458a65191 P 529e37385608459bb7d952862fa9e191 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:03:46.675488 9068 raft_consensus.cc:2258] T 221627d0921b4a808d037c7458a65191 P 529e37385608459bb7d952862fa9e191 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:03:46.676692 9069 ts_tablet_manager.cc:1905] T 221627d0921b4a808d037c7458a65191 P b873e04286174b76b4af16a01aea7cbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:03:46.677191 9068 ts_tablet_manager.cc:1905] T 221627d0921b4a808d037c7458a65191 P 529e37385608459bb7d952862fa9e191: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:03:46.684592 9068 ts_tablet_manager.cc:1918] T 221627d0921b4a808d037c7458a65191 P 529e37385608459bb7d952862fa9e191: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 07:03:46.684764 9068 log.cc:1196] T 221627d0921b4a808d037c7458a65191 P 529e37385608459bb7d952862fa9e191: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-2/wal/wals/221627d0921b4a808d037c7458a65191
I20260812 07:03:46.685307 9068 ts_tablet_manager.cc:1939] T 221627d0921b4a808d037c7458a65191 P 529e37385608459bb7d952862fa9e191: Deleting consensus metadata
I20260812 07:03:46.686956 9069 ts_tablet_manager.cc:1918] T 221627d0921b4a808d037c7458a65191 P b873e04286174b76b4af16a01aea7cbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 07:03:46.687160 9069 log.cc:1196] T 221627d0921b4a808d037c7458a65191 P b873e04286174b76b4af16a01aea7cbb: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-1/wal/wals/221627d0921b4a808d037c7458a65191
I20260812 07:03:46.687137 8412 catalog_manager.cc:4841] TS 529e37385608459bb7d952862fa9e191 (127.6.61.3:38361): tablet 221627d0921b4a808d037c7458a65191 (table test-table [id=7d5e460d3f1d4a488f0547a9cf34b9ff]) successfully deleted
I20260812 07:03:46.687506 9018 tablet_bootstrap.cc:492] T 92eb7400aa9a4d5587aff85cc50eccc7 P 529e37385608459bb7d952862fa9e191: No bootstrap required, opened a new log
I20260812 07:03:46.687700 9018 ts_tablet_manager.cc:1397] T 92eb7400aa9a4d5587aff85cc50eccc7 P 529e37385608459bb7d952862fa9e191: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.002s
I20260812 07:03:46.687752 9069 ts_tablet_manager.cc:1939] T 221627d0921b4a808d037c7458a65191 P b873e04286174b76b4af16a01aea7cbb: Deleting consensus metadata
I20260812 07:03:46.688513 9018 raft_consensus.cc:348] T 92eb7400aa9a4d5587aff85cc50eccc7 P 529e37385608459bb7d952862fa9e191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:46.688787 9018 raft_consensus.cc:374] T 92eb7400aa9a4d5587aff85cc50eccc7 P 529e37385608459bb7d952862fa9e191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:03:46.688881 9018 raft_consensus.cc:729] T 92eb7400aa9a4d5587aff85cc50eccc7 P 529e37385608459bb7d952862fa9e191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 529e37385608459bb7d952862fa9e191, State: Initialized, Role: FOLLOWER
I20260812 07:03:46.689513 8410 catalog_manager.cc:4841] TS b873e04286174b76b4af16a01aea7cbb (127.6.61.2:45959): tablet 221627d0921b4a808d037c7458a65191 (table test-table [id=7d5e460d3f1d4a488f0547a9cf34b9ff]) successfully deleted
I20260812 07:03:46.689164 9018 consensus_queue.cc:260] T 92eb7400aa9a4d5587aff85cc50eccc7 P 529e37385608459bb7d952862fa9e191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:46.690239 9018 ts_tablet_manager.cc:1428] T 92eb7400aa9a4d5587aff85cc50eccc7 P 529e37385608459bb7d952862fa9e191: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 07:03:46.824581 9024 raft_consensus.cc:482] T 92eb7400aa9a4d5587aff85cc50eccc7 P a6e3ffc000c840d1b0f8d22e592b556b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:03:46.824879 9024 raft_consensus.cc:504] T 92eb7400aa9a4d5587aff85cc50eccc7 P a6e3ffc000c840d1b0f8d22e592b556b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:46.826913 9024 leader_election.cc:288] T 92eb7400aa9a4d5587aff85cc50eccc7 P a6e3ffc000c840d1b0f8d22e592b556b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b873e04286174b76b4af16a01aea7cbb (127.6.61.2:45959), 529e37385608459bb7d952862fa9e191 (127.6.61.3:38361)
I20260812 07:03:46.844390 9043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:46.827985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:56777 (local address 127.6.61.2:45959)
0812 07:03:46.828226 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:46.828232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:46.835188 (+ 6956us) server_negotiation.cc:407] Connection header received
0812 07:03:46.835468 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:46.835475 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:46.835576 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:46.835711 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:03:46.837437 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.838656 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.841349 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.842810 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.842919 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:46.843032 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:46.843925 (+ 893us) server_negotiation.cc:299] Negotiation successful
0812 07:03:46.844073 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 07:03:46.845788 9075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:46.827115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:45959 (local address 127.6.61.1:56777)
0812 07:03:46.835028 (+ 7913us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:46.835059 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:46.835289 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:46.836215 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:46.836221 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:46.836253 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:03:46.836603 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:46.836617 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.838998 (+ 2381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.839005 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:46.841144 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:46.841162 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.843308 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.843315 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:46.843658 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:46.843715 (+ 57us) client_negotiation.cc:769] Sending connection context
0812 07:03:46.845091 (+ 1376us) client_negotiation.cc:240] Negotiation successful
0812 07:03:46.845261 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7726,"mutex_wait_us":40,"thread_start_us":14879,"threads_started":1}
I20260812 07:03:46.848398 8774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92eb7400aa9a4d5587aff85cc50eccc7" candidate_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b873e04286174b76b4af16a01aea7cbb" is_pre_election: true
I20260812 07:03:46.848963 8774 raft_consensus.cc:2454] T 92eb7400aa9a4d5587aff85cc50eccc7 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e3ffc000c840d1b0f8d22e592b556b in term 0.
I20260812 07:03:46.849694 8530 leader_election.cc:302] T 92eb7400aa9a4d5587aff85cc50eccc7 P a6e3ffc000c840d1b0f8d22e592b556b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e3ffc000c840d1b0f8d22e592b556b, b873e04286174b76b4af16a01aea7cbb; no voters:
I20260812 07:03:46.850157 9024 raft_consensus.cc:2781] T 92eb7400aa9a4d5587aff85cc50eccc7 P a6e3ffc000c840d1b0f8d22e592b556b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:03:46.850297 9024 raft_consensus.cc:482] T 92eb7400aa9a4d5587aff85cc50eccc7 P a6e3ffc000c840d1b0f8d22e592b556b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:03:46.850407 9024 raft_consensus.cc:3037] T 92eb7400aa9a4d5587aff85cc50eccc7 P a6e3ffc000c840d1b0f8d22e592b556b [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:03:46.852223 9076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:46.828467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:43335 (local address 127.6.61.3:38361)
0812 07:03:46.838396 (+ 9929us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:46.838405 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:46.842009 (+ 3604us) server_negotiation.cc:407] Connection header received
0812 07:03:46.842583 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:46.842590 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:46.842671 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:46.842743 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:03:46.843942 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.845347 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.848582 (+ 3235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.849975 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.850114 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:46.850280 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:46.851672 (+ 1392us) server_negotiation.cc:299] Negotiation successful
0812 07:03:46.851838 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9729,"thread_start_us":10311,"threads_started":1}
I20260812 07:03:46.853386 9074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:46.827507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:38361 (local address 127.6.61.1:43335)
0812 07:03:46.841902 (+ 14395us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:46.841924 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:46.842397 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:46.843047 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:46.843052 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:46.843127 (+ 75us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:03:46.843719 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:46.843735 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.846056 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.846062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:46.848391 (+ 2329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:46.848408 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.851193 (+ 2785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.851200 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:46.851412 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:46.851455 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 07:03:46.852796 (+ 1341us) client_negotiation.cc:240] Negotiation successful
0812 07:03:46.852952 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14219,"mutex_wait_us":1,"thread_start_us":7166,"threads_started":1}
I20260812 07:03:46.853739 9024 raft_consensus.cc:504] T 92eb7400aa9a4d5587aff85cc50eccc7 P a6e3ffc000c840d1b0f8d22e592b556b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:46.854346 8936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92eb7400aa9a4d5587aff85cc50eccc7" candidate_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529e37385608459bb7d952862fa9e191" is_pre_election: true
I20260812 07:03:46.854605 8936 raft_consensus.cc:2454] T 92eb7400aa9a4d5587aff85cc50eccc7 P 529e37385608459bb7d952862fa9e191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e3ffc000c840d1b0f8d22e592b556b in term 0.
I20260812 07:03:46.855115 8936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92eb7400aa9a4d5587aff85cc50eccc7" candidate_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529e37385608459bb7d952862fa9e191"
I20260812 07:03:46.855329 8936 raft_consensus.cc:3037] T 92eb7400aa9a4d5587aff85cc50eccc7 P 529e37385608459bb7d952862fa9e191 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:03:46.855408 9024 leader_election.cc:288] T 92eb7400aa9a4d5587aff85cc50eccc7 P a6e3ffc000c840d1b0f8d22e592b556b [CANDIDATE]: Term 1 election: Requested vote from peers b873e04286174b76b4af16a01aea7cbb (127.6.61.2:45959), 529e37385608459bb7d952862fa9e191 (127.6.61.3:38361)
I20260812 07:03:46.857504 8936 raft_consensus.cc:2454] T 92eb7400aa9a4d5587aff85cc50eccc7 P 529e37385608459bb7d952862fa9e191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e3ffc000c840d1b0f8d22e592b556b in term 1.
I20260812 07:03:46.857991 8533 leader_election.cc:302] T 92eb7400aa9a4d5587aff85cc50eccc7 P a6e3ffc000c840d1b0f8d22e592b556b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 529e37385608459bb7d952862fa9e191, a6e3ffc000c840d1b0f8d22e592b556b; no voters:
I20260812 07:03:46.858018 8774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92eb7400aa9a4d5587aff85cc50eccc7" candidate_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b873e04286174b76b4af16a01aea7cbb"
I20260812 07:03:46.858247 8774 raft_consensus.cc:3037] T 92eb7400aa9a4d5587aff85cc50eccc7 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:03:46.858349 9024 raft_consensus.cc:2781] T 92eb7400aa9a4d5587aff85cc50eccc7 P a6e3ffc000c840d1b0f8d22e592b556b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:03:46.863109 8774 raft_consensus.cc:2454] T 92eb7400aa9a4d5587aff85cc50eccc7 P b873e04286174b76b4af16a01aea7cbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e3ffc000c840d1b0f8d22e592b556b in term 1.
I20260812 07:03:46.864564 9024 raft_consensus.cc:686] T 92eb7400aa9a4d5587aff85cc50eccc7 P a6e3ffc000c840d1b0f8d22e592b556b [term 1 LEADER]: Becoming Leader. State: Replica: a6e3ffc000c840d1b0f8d22e592b556b, State: Running, Role: LEADER
I20260812 07:03:46.865058 9024 consensus_queue.cc:237] T 92eb7400aa9a4d5587aff85cc50eccc7 P a6e3ffc000c840d1b0f8d22e592b556b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:46.869228 8428 catalog_manager.cc:5495] T 92eb7400aa9a4d5587aff85cc50eccc7 P a6e3ffc000c840d1b0f8d22e592b556b reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e3ffc000c840d1b0f8d22e592b556b (127.6.61.1). New cstate: current_term: 1 leader_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } health_report { overall_health: UNKNOWN } } }
I20260812 07:03:46.915069 9080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:46.908114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37470 (local address 127.6.61.1:39565)
0812 07:03:46.908740 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:46.908744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:46.908764 (+ 20us) server_negotiation.cc:407] Connection header received
0812 07:03:46.908845 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:46.908848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:46.908920 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:46.909061 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:03:46.909561 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.910485 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.912596 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.913013 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.913111 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:46.913670 (+ 559us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:03:46.914361 (+ 691us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:03:46.914475 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:46.914633 (+ 158us) server_negotiation.cc:299] Negotiation successful
0812 07:03:46.914700 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":381,"threads_started":1}
I20260812 07:03:46.931427 9043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:46.925780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:51863 (local address 127.6.61.2:45959)
0812 07:03:46.926033 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:46.926040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:46.926454 (+ 414us) server_negotiation.cc:407] Connection header received
0812 07:03:46.926681 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:46.926685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:46.926778 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:46.926880 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:03:46.927396 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.928123 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.929509 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.930064 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.930207 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:46.930633 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:03:46.930745 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:03:46.930844 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:46.930983 (+ 139us) server_negotiation.cc:299] Negotiation successful
0812 07:03:46.931071 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260812 07:03:46.931499 9083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:46.925661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:45959 (local address 127.6.61.1:51863)
0812 07:03:46.926347 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:46.926363 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:46.926530 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:46.926904 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:46.926909 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:46.926922 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 07:03:46.927237 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:46.927248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.928270 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.928278 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:46.929371 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:46.929383 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.930247 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.930253 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:46.930394 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:46.930447 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 07:03:46.930846 (+ 399us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 07:03:46.930861 (+ 15us) client_negotiation.cc:769] Sending connection context
0812 07:03:46.930962 (+ 101us) client_negotiation.cc:240] Negotiation successful
0812 07:03:46.931101 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":306,"threads_started":1}
I20260812 07:03:46.946100 8774 raft_consensus.cc:1261] T 92eb7400aa9a4d5587aff85cc50eccc7 P b873e04286174b76b4af16a01aea7cbb [term 1 FOLLOWER]: Refusing update from remote peer a6e3ffc000c840d1b0f8d22e592b556b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:03:46.946235 8936 raft_consensus.cc:1261] T 92eb7400aa9a4d5587aff85cc50eccc7 P 529e37385608459bb7d952862fa9e191 [term 1 FOLLOWER]: Refusing update from remote peer a6e3ffc000c840d1b0f8d22e592b556b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:03:46.947068 9077 consensus_queue.cc:1035] T 92eb7400aa9a4d5587aff85cc50eccc7 P a6e3ffc000c840d1b0f8d22e592b556b [LEADER]: Connected to new peer: Peer: permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:03:46.947467 9024 consensus_queue.cc:1035] T 92eb7400aa9a4d5587aff85cc50eccc7 P a6e3ffc000c840d1b0f8d22e592b556b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:03:46.979465 9080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:46.972416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:48145 (local address 127.6.61.1:39565)
0812 07:03:46.972591 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:46.972598 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:46.972793 (+ 195us) server_negotiation.cc:407] Connection header received
0812 07:03:46.973003 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:46.973009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:46.973088 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:46.973188 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:03:46.973574 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.974679 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.977252 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.977918 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.978043 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:46.978639 (+ 596us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:03:46.978781 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:03:46.978900 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:46.979035 (+ 135us) server_negotiation.cc:299] Negotiation successful
0812 07:03:46.979122 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 07:03:46.981323 9058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:46.972502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:39565 (local address 127.6.61.2:48145)
0812 07:03:46.972697 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:46.972713 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:46.972862 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:46.973206 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:46.973211 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:46.973227 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 07:03:46.973432 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:46.973440 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.974867 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.974873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:46.976575 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:46.976593 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:46.978211 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:46.978218 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:46.978404 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:46.978461 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 07:03:46.978904 (+ 443us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 07:03:46.978919 (+ 15us) client_negotiation.cc:769] Sending connection context
0812 07:03:46.979013 (+ 94us) client_negotiation.cc:240] Negotiation successful
0812 07:03:46.980908 (+ 1895us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
W20260812 07:03:46.982694 9045 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 07:03:48.590509 8428 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:03:48.590786 8428 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:03:48.594895 8428 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 100f02a2c2c24a9f9f301631ca439064: Sending DeleteTablet for 3 replicas of tablet 92eb7400aa9a4d5587aff85cc50eccc7
I20260812 07:03:48.595706 8585 tablet_service.cc:1514] Processing DeleteTablet for tablet 92eb7400aa9a4d5587aff85cc50eccc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:03:48 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:37468
I20260812 07:03:48.595945 8752 tablet_service.cc:1514] Processing DeleteTablet for tablet 92eb7400aa9a4d5587aff85cc50eccc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:03:48 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:46138
I20260812 07:03:48.595961 8913 tablet_service.cc:1514] Processing DeleteTablet for tablet 92eb7400aa9a4d5587aff85cc50eccc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:03:48 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:43640
I20260812 07:03:48.606812 9115 tablet_replica.cc:330] T 92eb7400aa9a4d5587aff85cc50eccc7 P b873e04286174b76b4af16a01aea7cbb: stopping tablet replica
I20260812 07:03:48.607224 9115 raft_consensus.cc:2229] T 92eb7400aa9a4d5587aff85cc50eccc7 P b873e04286174b76b4af16a01aea7cbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:03:48.607643 9115 raft_consensus.cc:2258] T 92eb7400aa9a4d5587aff85cc50eccc7 P b873e04286174b76b4af16a01aea7cbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:03:48.609138 9115 ts_tablet_manager.cc:1905] T 92eb7400aa9a4d5587aff85cc50eccc7 P b873e04286174b76b4af16a01aea7cbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:03:48.611136 9114 tablet_replica.cc:330] T 92eb7400aa9a4d5587aff85cc50eccc7 P a6e3ffc000c840d1b0f8d22e592b556b: stopping tablet replica
I20260812 07:03:48.611310 9116 tablet_replica.cc:330] T 92eb7400aa9a4d5587aff85cc50eccc7 P 529e37385608459bb7d952862fa9e191: stopping tablet replica
I20260812 07:03:48.611383 9114 raft_consensus.cc:2229] T 92eb7400aa9a4d5587aff85cc50eccc7 P a6e3ffc000c840d1b0f8d22e592b556b [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:03:48.611528 9116 raft_consensus.cc:2229] T 92eb7400aa9a4d5587aff85cc50eccc7 P 529e37385608459bb7d952862fa9e191 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:03:48.611784 9116 raft_consensus.cc:2258] T 92eb7400aa9a4d5587aff85cc50eccc7 P 529e37385608459bb7d952862fa9e191 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:03:48.612037 9114 raft_consensus.cc:2258] T 92eb7400aa9a4d5587aff85cc50eccc7 P a6e3ffc000c840d1b0f8d22e592b556b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:03:48.612886 9116 ts_tablet_manager.cc:1905] T 92eb7400aa9a4d5587aff85cc50eccc7 P 529e37385608459bb7d952862fa9e191: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:03:48.613077 9114 ts_tablet_manager.cc:1905] T 92eb7400aa9a4d5587aff85cc50eccc7 P a6e3ffc000c840d1b0f8d22e592b556b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:03:48.617126 9115 ts_tablet_manager.cc:1918] T 92eb7400aa9a4d5587aff85cc50eccc7 P b873e04286174b76b4af16a01aea7cbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 07:03:48.617331 9115 log.cc:1196] T 92eb7400aa9a4d5587aff85cc50eccc7 P b873e04286174b76b4af16a01aea7cbb: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-1/wal/wals/92eb7400aa9a4d5587aff85cc50eccc7
I20260812 07:03:48.617956 9115 ts_tablet_manager.cc:1939] T 92eb7400aa9a4d5587aff85cc50eccc7 P b873e04286174b76b4af16a01aea7cbb: Deleting consensus metadata
I20260812 07:03:48.619738 8410 catalog_manager.cc:4841] TS b873e04286174b76b4af16a01aea7cbb (127.6.61.2:45959): tablet 92eb7400aa9a4d5587aff85cc50eccc7 (table test-table [id=45d7c8ba52404d7793311350110ac8f9]) successfully deleted
I20260812 07:03:48.619801 9116 ts_tablet_manager.cc:1918] T 92eb7400aa9a4d5587aff85cc50eccc7 P 529e37385608459bb7d952862fa9e191: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 07:03:48.619923 9116 log.cc:1196] T 92eb7400aa9a4d5587aff85cc50eccc7 P 529e37385608459bb7d952862fa9e191: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-2/wal/wals/92eb7400aa9a4d5587aff85cc50eccc7
I20260812 07:03:48.620383 9116 ts_tablet_manager.cc:1939] T 92eb7400aa9a4d5587aff85cc50eccc7 P 529e37385608459bb7d952862fa9e191: Deleting consensus metadata
I20260812 07:03:48.620916 9114 ts_tablet_manager.cc:1918] T 92eb7400aa9a4d5587aff85cc50eccc7 P a6e3ffc000c840d1b0f8d22e592b556b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 07:03:48.621063 9114 log.cc:1196] T 92eb7400aa9a4d5587aff85cc50eccc7 P a6e3ffc000c840d1b0f8d22e592b556b: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-0/wal/wals/92eb7400aa9a4d5587aff85cc50eccc7
I20260812 07:03:48.621464 8412 catalog_manager.cc:4841] TS 529e37385608459bb7d952862fa9e191 (127.6.61.3:38361): tablet 92eb7400aa9a4d5587aff85cc50eccc7 (table test-table [id=45d7c8ba52404d7793311350110ac8f9]) successfully deleted
I20260812 07:03:48.621555 9114 ts_tablet_manager.cc:1939] T 92eb7400aa9a4d5587aff85cc50eccc7 P a6e3ffc000c840d1b0f8d22e592b556b: Deleting consensus metadata
I20260812 07:03:48.622776 8408 catalog_manager.cc:4841] TS a6e3ffc000c840d1b0f8d22e592b556b (127.6.61.1:39565): tablet 92eb7400aa9a4d5587aff85cc50eccc7 (table test-table [id=45d7c8ba52404d7793311350110ac8f9]) successfully deleted
Aug 12 07:03:48 dist-test-slave-69rg krb5kdc[8377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518228, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 07:03:48 dist-test-slave-69rg krb5kdc[8377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518228, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:03:48.666754 9124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:48.651576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54338 (local address 127.6.61.62:39233)
0812 07:03:48.652290 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:48.652299 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:48.652328 (+ 29us) server_negotiation.cc:407] Connection header received
0812 07:03:48.652563 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:48.652567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:48.652635 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:48.652772 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:48.654256 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:48.655065 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:48.656336 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:48.656750 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:48.656889 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:48.661239 (+ 4350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:48.661244 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:48.661247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:48.661282 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:48.664045 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:48.664752 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:48.664759 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:48.664762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:48.664904 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:48.665280 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:48.665285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:48.665288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:48.665578 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:48.665866 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:48.666175 (+ 309us) server_negotiation.cc:299] Negotiation successful
0812 07:03:48.666398 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":397,"threads_started":1}
I20260812 07:03:48.670634 8428 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:03:48.671324 8428 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:03:48.678537 8585 tablet_service.cc:1467] Processing CreateTablet for tablet b82bddde32774576897ddaac0db57ee3 (DEFAULT_TABLE table=test-table [id=941f29c1edbb4c128e8758b7651bbccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:03:48.678629 8752 tablet_service.cc:1467] Processing CreateTablet for tablet b82bddde32774576897ddaac0db57ee3 (DEFAULT_TABLE table=test-table [id=941f29c1edbb4c128e8758b7651bbccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:03:48.678910 8913 tablet_service.cc:1467] Processing CreateTablet for tablet b82bddde32774576897ddaac0db57ee3 (DEFAULT_TABLE table=test-table [id=941f29c1edbb4c128e8758b7651bbccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:03:48.679095 8752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82bddde32774576897ddaac0db57ee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:03:48.679097 8585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82bddde32774576897ddaac0db57ee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:03:48.679332 8913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82bddde32774576897ddaac0db57ee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:03:48.685521 9125 tablet_bootstrap.cc:492] T b82bddde32774576897ddaac0db57ee3 P a6e3ffc000c840d1b0f8d22e592b556b: Bootstrap starting.
I20260812 07:03:48.687427 9125 tablet_bootstrap.cc:654] T b82bddde32774576897ddaac0db57ee3 P a6e3ffc000c840d1b0f8d22e592b556b: Neither blocks nor log segments found. Creating new log.
I20260812 07:03:48.688872 9128 tablet_bootstrap.cc:492] T b82bddde32774576897ddaac0db57ee3 P b873e04286174b76b4af16a01aea7cbb: Bootstrap starting.
I20260812 07:03:48.690356 9127 tablet_bootstrap.cc:492] T b82bddde32774576897ddaac0db57ee3 P 529e37385608459bb7d952862fa9e191: Bootstrap starting.
I20260812 07:03:48.691537 9128 tablet_bootstrap.cc:654] T b82bddde32774576897ddaac0db57ee3 P b873e04286174b76b4af16a01aea7cbb: Neither blocks nor log segments found. Creating new log.
I20260812 07:03:48.691963 9125 tablet_bootstrap.cc:492] T b82bddde32774576897ddaac0db57ee3 P a6e3ffc000c840d1b0f8d22e592b556b: No bootstrap required, opened a new log
I20260812 07:03:48.692133 9125 ts_tablet_manager.cc:1397] T b82bddde32774576897ddaac0db57ee3 P a6e3ffc000c840d1b0f8d22e592b556b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260812 07:03:48.692204 9127 tablet_bootstrap.cc:654] T b82bddde32774576897ddaac0db57ee3 P 529e37385608459bb7d952862fa9e191: Neither blocks nor log segments found. Creating new log.
I20260812 07:03:48.693009 9125 raft_consensus.cc:348] T b82bddde32774576897ddaac0db57ee3 P a6e3ffc000c840d1b0f8d22e592b556b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:48.693346 9125 raft_consensus.cc:374] T b82bddde32774576897ddaac0db57ee3 P a6e3ffc000c840d1b0f8d22e592b556b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:03:48.693451 9125 raft_consensus.cc:729] T b82bddde32774576897ddaac0db57ee3 P a6e3ffc000c840d1b0f8d22e592b556b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e3ffc000c840d1b0f8d22e592b556b, State: Initialized, Role: FOLLOWER
I20260812 07:03:48.693748 9125 consensus_queue.cc:260] T b82bddde32774576897ddaac0db57ee3 P a6e3ffc000c840d1b0f8d22e592b556b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:48.694141 9128 tablet_bootstrap.cc:492] T b82bddde32774576897ddaac0db57ee3 P b873e04286174b76b4af16a01aea7cbb: No bootstrap required, opened a new log
I20260812 07:03:48.694299 9128 ts_tablet_manager.cc:1397] T b82bddde32774576897ddaac0db57ee3 P b873e04286174b76b4af16a01aea7cbb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260812 07:03:48.694439 9125 ts_tablet_manager.cc:1428] T b82bddde32774576897ddaac0db57ee3 P a6e3ffc000c840d1b0f8d22e592b556b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 07:03:48.695106 9127 tablet_bootstrap.cc:492] T b82bddde32774576897ddaac0db57ee3 P 529e37385608459bb7d952862fa9e191: No bootstrap required, opened a new log
I20260812 07:03:48.695272 9127 ts_tablet_manager.cc:1397] T b82bddde32774576897ddaac0db57ee3 P 529e37385608459bb7d952862fa9e191: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260812 07:03:48.695184 9128 raft_consensus.cc:348] T b82bddde32774576897ddaac0db57ee3 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:48.695461 9128 raft_consensus.cc:374] T b82bddde32774576897ddaac0db57ee3 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:03:48.695581 9128 raft_consensus.cc:729] T b82bddde32774576897ddaac0db57ee3 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b873e04286174b76b4af16a01aea7cbb, State: Initialized, Role: FOLLOWER
I20260812 07:03:48.695864 9128 consensus_queue.cc:260] T b82bddde32774576897ddaac0db57ee3 P b873e04286174b76b4af16a01aea7cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:48.696126 9127 raft_consensus.cc:348] T b82bddde32774576897ddaac0db57ee3 P 529e37385608459bb7d952862fa9e191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:48.696399 9127 raft_consensus.cc:374] T b82bddde32774576897ddaac0db57ee3 P 529e37385608459bb7d952862fa9e191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:03:48.696506 9127 raft_consensus.cc:729] T b82bddde32774576897ddaac0db57ee3 P 529e37385608459bb7d952862fa9e191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 529e37385608459bb7d952862fa9e191, State: Initialized, Role: FOLLOWER
I20260812 07:03:48.696552 9128 ts_tablet_manager.cc:1428] T b82bddde32774576897ddaac0db57ee3 P b873e04286174b76b4af16a01aea7cbb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 07:03:48.696799 9127 consensus_queue.cc:260] T b82bddde32774576897ddaac0db57ee3 P 529e37385608459bb7d952862fa9e191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:48.698745 9127 ts_tablet_manager.cc:1428] T b82bddde32774576897ddaac0db57ee3 P 529e37385608459bb7d952862fa9e191: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260812 07:03:48.755393 9098 raft_consensus.cc:482] T b82bddde32774576897ddaac0db57ee3 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:03:48.755630 9098 raft_consensus.cc:504] T b82bddde32774576897ddaac0db57ee3 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:48.756368 9098 leader_election.cc:288] T b82bddde32774576897ddaac0db57ee3 P b873e04286174b76b4af16a01aea7cbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e3ffc000c840d1b0f8d22e592b556b (127.6.61.1:39565), 529e37385608459bb7d952862fa9e191 (127.6.61.3:38361)
I20260812 07:03:48.756932 8611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82bddde32774576897ddaac0db57ee3" candidate_uuid: "b873e04286174b76b4af16a01aea7cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" is_pre_election: true
I20260812 07:03:48.757009 8936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82bddde32774576897ddaac0db57ee3" candidate_uuid: "b873e04286174b76b4af16a01aea7cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529e37385608459bb7d952862fa9e191" is_pre_election: true
I20260812 07:03:48.757244 8936 raft_consensus.cc:2454] T b82bddde32774576897ddaac0db57ee3 P 529e37385608459bb7d952862fa9e191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b873e04286174b76b4af16a01aea7cbb in term 0.
I20260812 07:03:48.757210 8611 raft_consensus.cc:2454] T b82bddde32774576897ddaac0db57ee3 P a6e3ffc000c840d1b0f8d22e592b556b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b873e04286174b76b4af16a01aea7cbb in term 0.
I20260812 07:03:48.757735 8696 leader_election.cc:302] T b82bddde32774576897ddaac0db57ee3 P b873e04286174b76b4af16a01aea7cbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 529e37385608459bb7d952862fa9e191, b873e04286174b76b4af16a01aea7cbb; no voters:
I20260812 07:03:48.758051 9098 raft_consensus.cc:2781] T b82bddde32774576897ddaac0db57ee3 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:03:48.758198 9098 raft_consensus.cc:482] T b82bddde32774576897ddaac0db57ee3 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:03:48.758317 9098 raft_consensus.cc:3037] T b82bddde32774576897ddaac0db57ee3 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:03:48.760500 9098 raft_consensus.cc:504] T b82bddde32774576897ddaac0db57ee3 P b873e04286174b76b4af16a01aea7cbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:48.761096 9098 leader_election.cc:288] T b82bddde32774576897ddaac0db57ee3 P b873e04286174b76b4af16a01aea7cbb [CANDIDATE]: Term 1 election: Requested vote from peers a6e3ffc000c840d1b0f8d22e592b556b (127.6.61.1:39565), 529e37385608459bb7d952862fa9e191 (127.6.61.3:38361)
I20260812 07:03:48.761552 8611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82bddde32774576897ddaac0db57ee3" candidate_uuid: "b873e04286174b76b4af16a01aea7cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e3ffc000c840d1b0f8d22e592b556b"
I20260812 07:03:48.761587 8936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82bddde32774576897ddaac0db57ee3" candidate_uuid: "b873e04286174b76b4af16a01aea7cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529e37385608459bb7d952862fa9e191"
I20260812 07:03:48.761790 8611 raft_consensus.cc:3037] T b82bddde32774576897ddaac0db57ee3 P a6e3ffc000c840d1b0f8d22e592b556b [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:03:48.761847 8936 raft_consensus.cc:3037] T b82bddde32774576897ddaac0db57ee3 P 529e37385608459bb7d952862fa9e191 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:03:48.764494 8936 raft_consensus.cc:2454] T b82bddde32774576897ddaac0db57ee3 P 529e37385608459bb7d952862fa9e191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b873e04286174b76b4af16a01aea7cbb in term 1.
I20260812 07:03:48.764909 8611 raft_consensus.cc:2454] T b82bddde32774576897ddaac0db57ee3 P a6e3ffc000c840d1b0f8d22e592b556b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b873e04286174b76b4af16a01aea7cbb in term 1.
I20260812 07:03:48.764968 8696 leader_election.cc:302] T b82bddde32774576897ddaac0db57ee3 P b873e04286174b76b4af16a01aea7cbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 529e37385608459bb7d952862fa9e191, b873e04286174b76b4af16a01aea7cbb; no voters:
I20260812 07:03:48.765264 9098 raft_consensus.cc:2781] T b82bddde32774576897ddaac0db57ee3 P b873e04286174b76b4af16a01aea7cbb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:03:48.765434 9098 raft_consensus.cc:686] T b82bddde32774576897ddaac0db57ee3 P b873e04286174b76b4af16a01aea7cbb [term 1 LEADER]: Becoming Leader. State: Replica: b873e04286174b76b4af16a01aea7cbb, State: Running, Role: LEADER
I20260812 07:03:48.765736 9098 consensus_queue.cc:237] T b82bddde32774576897ddaac0db57ee3 P b873e04286174b76b4af16a01aea7cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:48.768399 8427 catalog_manager.cc:5495] T b82bddde32774576897ddaac0db57ee3 P b873e04286174b76b4af16a01aea7cbb reported cstate change: term changed from 0 to 1, leader changed from <none> to b873e04286174b76b4af16a01aea7cbb (127.6.61.2). New cstate: current_term: 1 leader_uuid: "b873e04286174b76b4af16a01aea7cbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } health_report { overall_health: UNKNOWN } } }
I20260812 07:03:48.801736 9136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:48.796892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46160 (local address 127.6.61.2:45959)
0812 07:03:48.797523 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:48.797528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:48.797547 (+ 19us) server_negotiation.cc:407] Connection header received
0812 07:03:48.797599 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:48.797602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:48.797661 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:48.797746 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:03:48.798172 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:48.798954 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:48.800214 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:48.800624 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:48.800718 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:48.801076 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:03:48.801178 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:03:48.801246 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:48.801405 (+ 159us) server_negotiation.cc:299] Negotiation successful
0812 07:03:48.801464 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":365,"threads_started":1}
I20260812 07:03:48.810930 8936 raft_consensus.cc:1261] T b82bddde32774576897ddaac0db57ee3 P 529e37385608459bb7d952862fa9e191 [term 1 FOLLOWER]: Refusing update from remote peer b873e04286174b76b4af16a01aea7cbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:03:48.810945 8611 raft_consensus.cc:1261] T b82bddde32774576897ddaac0db57ee3 P a6e3ffc000c840d1b0f8d22e592b556b [term 1 FOLLOWER]: Refusing update from remote peer b873e04286174b76b4af16a01aea7cbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:03:48.811513 9023 consensus_queue.cc:1035] T b82bddde32774576897ddaac0db57ee3 P b873e04286174b76b4af16a01aea7cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:03:48.811784 9093 consensus_queue.cc:1035] T b82bddde32774576897ddaac0db57ee3 P b873e04286174b76b4af16a01aea7cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:03:48.864163 8427 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:03:48.864455 8427 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:03:48.867439 8427 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 100f02a2c2c24a9f9f301631ca439064: Sending DeleteTablet for 3 replicas of tablet b82bddde32774576897ddaac0db57ee3
I20260812 07:03:48.868250 8585 tablet_service.cc:1514] Processing DeleteTablet for tablet b82bddde32774576897ddaac0db57ee3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:03:48 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:37468
I20260812 07:03:48.868335 8913 tablet_service.cc:1514] Processing DeleteTablet for tablet b82bddde32774576897ddaac0db57ee3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:03:48 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:43640
I20260812 07:03:48.868570 9114 tablet_replica.cc:330] T b82bddde32774576897ddaac0db57ee3 P a6e3ffc000c840d1b0f8d22e592b556b: stopping tablet replica
I20260812 07:03:48.868613 9116 tablet_replica.cc:330] T b82bddde32774576897ddaac0db57ee3 P 529e37385608459bb7d952862fa9e191: stopping tablet replica
I20260812 07:03:48.868662 8752 tablet_service.cc:1514] Processing DeleteTablet for tablet b82bddde32774576897ddaac0db57ee3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:03:48 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:46138
I20260812 07:03:48.868793 9116 raft_consensus.cc:2229] T b82bddde32774576897ddaac0db57ee3 P 529e37385608459bb7d952862fa9e191 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:03:48.868798 9114 raft_consensus.cc:2229] T b82bddde32774576897ddaac0db57ee3 P a6e3ffc000c840d1b0f8d22e592b556b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:03:48.868947 9115 tablet_replica.cc:330] T b82bddde32774576897ddaac0db57ee3 P b873e04286174b76b4af16a01aea7cbb: stopping tablet replica
I20260812 07:03:48.868983 9116 raft_consensus.cc:2258] T b82bddde32774576897ddaac0db57ee3 P 529e37385608459bb7d952862fa9e191 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:03:48.868983 9114 raft_consensus.cc:2258] T b82bddde32774576897ddaac0db57ee3 P a6e3ffc000c840d1b0f8d22e592b556b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:03:48.869153 9115 raft_consensus.cc:2229] T b82bddde32774576897ddaac0db57ee3 P b873e04286174b76b4af16a01aea7cbb [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:03:48.869509 9115 raft_consensus.cc:2258] T b82bddde32774576897ddaac0db57ee3 P b873e04286174b76b4af16a01aea7cbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:03:48.869654 8427 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:03:48.870419 8427 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:03:48.870488 9116 ts_tablet_manager.cc:1905] T b82bddde32774576897ddaac0db57ee3 P 529e37385608459bb7d952862fa9e191: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:03:48.870895 9115 ts_tablet_manager.cc:1905] T b82bddde32774576897ddaac0db57ee3 P b873e04286174b76b4af16a01aea7cbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:03:48.871721 9114 ts_tablet_manager.cc:1905] T b82bddde32774576897ddaac0db57ee3 P a6e3ffc000c840d1b0f8d22e592b556b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:03:48.876436 9116 ts_tablet_manager.cc:1918] T b82bddde32774576897ddaac0db57ee3 P 529e37385608459bb7d952862fa9e191: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 07:03:48.876600 9116 log.cc:1196] T b82bddde32774576897ddaac0db57ee3 P 529e37385608459bb7d952862fa9e191: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-2/wal/wals/b82bddde32774576897ddaac0db57ee3
I20260812 07:03:48.877049 9116 ts_tablet_manager.cc:1939] T b82bddde32774576897ddaac0db57ee3 P 529e37385608459bb7d952862fa9e191: Deleting consensus metadata
I20260812 07:03:48.878360 8412 catalog_manager.cc:4841] TS 529e37385608459bb7d952862fa9e191 (127.6.61.3:38361): tablet b82bddde32774576897ddaac0db57ee3 (table test-table [id=941f29c1edbb4c128e8758b7651bbccd]) successfully deleted
I20260812 07:03:48.879484 8585 tablet_service.cc:1467] Processing CreateTablet for tablet 39da7d3ad5cd4134acd4ff11257b5487 (DEFAULT_TABLE table=test-table [id=fc44146d04104b449bb28e22534064bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:03:48.879664 8752 tablet_service.cc:1467] Processing CreateTablet for tablet 39da7d3ad5cd4134acd4ff11257b5487 (DEFAULT_TABLE table=test-table [id=fc44146d04104b449bb28e22534064bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:03:48.879791 9115 ts_tablet_manager.cc:1918] T b82bddde32774576897ddaac0db57ee3 P b873e04286174b76b4af16a01aea7cbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 07:03:48.879774 8913 tablet_service.cc:1467] Processing CreateTablet for tablet 39da7d3ad5cd4134acd4ff11257b5487 (DEFAULT_TABLE table=test-table [id=fc44146d04104b449bb28e22534064bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:03:48.879914 8585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39da7d3ad5cd4134acd4ff11257b5487. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:03:48.879927 9115 log.cc:1196] T b82bddde32774576897ddaac0db57ee3 P b873e04286174b76b4af16a01aea7cbb: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-1/wal/wals/b82bddde32774576897ddaac0db57ee3
I20260812 07:03:48.880118 8752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39da7d3ad5cd4134acd4ff11257b5487. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:03:48.880187 8913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39da7d3ad5cd4134acd4ff11257b5487. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:03:48.880450 9115 ts_tablet_manager.cc:1939] T b82bddde32774576897ddaac0db57ee3 P b873e04286174b76b4af16a01aea7cbb: Deleting consensus metadata
I20260812 07:03:48.884749 9128 tablet_bootstrap.cc:492] T 39da7d3ad5cd4134acd4ff11257b5487 P b873e04286174b76b4af16a01aea7cbb: Bootstrap starting.
I20260812 07:03:48.885088 8410 catalog_manager.cc:4841] TS b873e04286174b76b4af16a01aea7cbb (127.6.61.2:45959): tablet b82bddde32774576897ddaac0db57ee3 (table test-table [id=941f29c1edbb4c128e8758b7651bbccd]) successfully deleted
I20260812 07:03:48.886363 9125 tablet_bootstrap.cc:492] T 39da7d3ad5cd4134acd4ff11257b5487 P a6e3ffc000c840d1b0f8d22e592b556b: Bootstrap starting.
I20260812 07:03:48.886430 9128 tablet_bootstrap.cc:654] T 39da7d3ad5cd4134acd4ff11257b5487 P b873e04286174b76b4af16a01aea7cbb: Neither blocks nor log segments found. Creating new log.
I20260812 07:03:48.886904 9127 tablet_bootstrap.cc:492] T 39da7d3ad5cd4134acd4ff11257b5487 P 529e37385608459bb7d952862fa9e191: Bootstrap starting.
I20260812 07:03:48.888386 9125 tablet_bootstrap.cc:654] T 39da7d3ad5cd4134acd4ff11257b5487 P a6e3ffc000c840d1b0f8d22e592b556b: Neither blocks nor log segments found. Creating new log.
I20260812 07:03:48.888499 9127 tablet_bootstrap.cc:654] T 39da7d3ad5cd4134acd4ff11257b5487 P 529e37385608459bb7d952862fa9e191: Neither blocks nor log segments found. Creating new log.
I20260812 07:03:48.888844 9114 ts_tablet_manager.cc:1918] T b82bddde32774576897ddaac0db57ee3 P a6e3ffc000c840d1b0f8d22e592b556b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 07:03:48.889070 9114 log.cc:1196] T b82bddde32774576897ddaac0db57ee3 P a6e3ffc000c840d1b0f8d22e592b556b: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-0/wal/wals/b82bddde32774576897ddaac0db57ee3
I20260812 07:03:48.889740 9114 ts_tablet_manager.cc:1939] T b82bddde32774576897ddaac0db57ee3 P a6e3ffc000c840d1b0f8d22e592b556b: Deleting consensus metadata
I20260812 07:03:48.891130 8408 catalog_manager.cc:4841] TS a6e3ffc000c840d1b0f8d22e592b556b (127.6.61.1:39565): tablet b82bddde32774576897ddaac0db57ee3 (table test-table [id=941f29c1edbb4c128e8758b7651bbccd]) successfully deleted
I20260812 07:03:48.891990 9127 tablet_bootstrap.cc:492] T 39da7d3ad5cd4134acd4ff11257b5487 P 529e37385608459bb7d952862fa9e191: No bootstrap required, opened a new log
I20260812 07:03:48.892184 9127 ts_tablet_manager.cc:1397] T 39da7d3ad5cd4134acd4ff11257b5487 P 529e37385608459bb7d952862fa9e191: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260812 07:03:48.892968 9127 raft_consensus.cc:348] T 39da7d3ad5cd4134acd4ff11257b5487 P 529e37385608459bb7d952862fa9e191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:48.893246 9127 raft_consensus.cc:374] T 39da7d3ad5cd4134acd4ff11257b5487 P 529e37385608459bb7d952862fa9e191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:03:48.893347 9127 raft_consensus.cc:729] T 39da7d3ad5cd4134acd4ff11257b5487 P 529e37385608459bb7d952862fa9e191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 529e37385608459bb7d952862fa9e191, State: Initialized, Role: FOLLOWER
I20260812 07:03:48.893648 9127 consensus_queue.cc:260] T 39da7d3ad5cd4134acd4ff11257b5487 P 529e37385608459bb7d952862fa9e191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:48.894371 9127 ts_tablet_manager.cc:1428] T 39da7d3ad5cd4134acd4ff11257b5487 P 529e37385608459bb7d952862fa9e191: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 07:03:48.899785 9128 tablet_bootstrap.cc:492] T 39da7d3ad5cd4134acd4ff11257b5487 P b873e04286174b76b4af16a01aea7cbb: No bootstrap required, opened a new log
I20260812 07:03:48.899792 9125 tablet_bootstrap.cc:492] T 39da7d3ad5cd4134acd4ff11257b5487 P a6e3ffc000c840d1b0f8d22e592b556b: No bootstrap required, opened a new log
I20260812 07:03:48.899963 9128 ts_tablet_manager.cc:1397] T 39da7d3ad5cd4134acd4ff11257b5487 P b873e04286174b76b4af16a01aea7cbb: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.004s
I20260812 07:03:48.899978 9125 ts_tablet_manager.cc:1397] T 39da7d3ad5cd4134acd4ff11257b5487 P a6e3ffc000c840d1b0f8d22e592b556b: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.005s
I20260812 07:03:48.900785 9125 raft_consensus.cc:348] T 39da7d3ad5cd4134acd4ff11257b5487 P a6e3ffc000c840d1b0f8d22e592b556b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:48.900779 9128 raft_consensus.cc:348] T 39da7d3ad5cd4134acd4ff11257b5487 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:48.901026 9125 raft_consensus.cc:374] T 39da7d3ad5cd4134acd4ff11257b5487 P a6e3ffc000c840d1b0f8d22e592b556b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:03:48.901029 9128 raft_consensus.cc:374] T 39da7d3ad5cd4134acd4ff11257b5487 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:03:48.901125 9128 raft_consensus.cc:729] T 39da7d3ad5cd4134acd4ff11257b5487 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b873e04286174b76b4af16a01aea7cbb, State: Initialized, Role: FOLLOWER
I20260812 07:03:48.901126 9125 raft_consensus.cc:729] T 39da7d3ad5cd4134acd4ff11257b5487 P a6e3ffc000c840d1b0f8d22e592b556b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e3ffc000c840d1b0f8d22e592b556b, State: Initialized, Role: FOLLOWER
I20260812 07:03:48.901379 9128 consensus_queue.cc:260] T 39da7d3ad5cd4134acd4ff11257b5487 P b873e04286174b76b4af16a01aea7cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:48.901386 9125 consensus_queue.cc:260] T 39da7d3ad5cd4134acd4ff11257b5487 P a6e3ffc000c840d1b0f8d22e592b556b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:48.902015 9125 ts_tablet_manager.cc:1428] T 39da7d3ad5cd4134acd4ff11257b5487 P a6e3ffc000c840d1b0f8d22e592b556b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 07:03:48.902439 9128 ts_tablet_manager.cc:1428] T 39da7d3ad5cd4134acd4ff11257b5487 P b873e04286174b76b4af16a01aea7cbb: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260812 07:03:49.119499 9093 raft_consensus.cc:482] T 39da7d3ad5cd4134acd4ff11257b5487 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:03:49.119789 9093 raft_consensus.cc:504] T 39da7d3ad5cd4134acd4ff11257b5487 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:49.120484 9093 leader_election.cc:288] T 39da7d3ad5cd4134acd4ff11257b5487 P b873e04286174b76b4af16a01aea7cbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e3ffc000c840d1b0f8d22e592b556b (127.6.61.1:39565), 529e37385608459bb7d952862fa9e191 (127.6.61.3:38361)
I20260812 07:03:49.121153 8935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39da7d3ad5cd4134acd4ff11257b5487" candidate_uuid: "b873e04286174b76b4af16a01aea7cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529e37385608459bb7d952862fa9e191" is_pre_election: true
I20260812 07:03:49.121193 8611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39da7d3ad5cd4134acd4ff11257b5487" candidate_uuid: "b873e04286174b76b4af16a01aea7cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" is_pre_election: true
I20260812 07:03:49.121440 8935 raft_consensus.cc:2454] T 39da7d3ad5cd4134acd4ff11257b5487 P 529e37385608459bb7d952862fa9e191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b873e04286174b76b4af16a01aea7cbb in term 0.
I20260812 07:03:49.121531 8611 raft_consensus.cc:2454] T 39da7d3ad5cd4134acd4ff11257b5487 P a6e3ffc000c840d1b0f8d22e592b556b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b873e04286174b76b4af16a01aea7cbb in term 0.
I20260812 07:03:49.122010 8696 leader_election.cc:302] T 39da7d3ad5cd4134acd4ff11257b5487 P b873e04286174b76b4af16a01aea7cbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 529e37385608459bb7d952862fa9e191, b873e04286174b76b4af16a01aea7cbb; no voters:
I20260812 07:03:49.122357 9093 raft_consensus.cc:2781] T 39da7d3ad5cd4134acd4ff11257b5487 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:03:49.122469 9093 raft_consensus.cc:482] T 39da7d3ad5cd4134acd4ff11257b5487 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:03:49.122577 9093 raft_consensus.cc:3037] T 39da7d3ad5cd4134acd4ff11257b5487 P b873e04286174b76b4af16a01aea7cbb [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:03:49.124949 9093 raft_consensus.cc:504] T 39da7d3ad5cd4134acd4ff11257b5487 P b873e04286174b76b4af16a01aea7cbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:49.125559 9093 leader_election.cc:288] T 39da7d3ad5cd4134acd4ff11257b5487 P b873e04286174b76b4af16a01aea7cbb [CANDIDATE]: Term 1 election: Requested vote from peers a6e3ffc000c840d1b0f8d22e592b556b (127.6.61.1:39565), 529e37385608459bb7d952862fa9e191 (127.6.61.3:38361)
I20260812 07:03:49.126073 8611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39da7d3ad5cd4134acd4ff11257b5487" candidate_uuid: "b873e04286174b76b4af16a01aea7cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e3ffc000c840d1b0f8d22e592b556b"
I20260812 07:03:49.126178 8935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39da7d3ad5cd4134acd4ff11257b5487" candidate_uuid: "b873e04286174b76b4af16a01aea7cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529e37385608459bb7d952862fa9e191"
I20260812 07:03:49.126318 8611 raft_consensus.cc:3037] T 39da7d3ad5cd4134acd4ff11257b5487 P a6e3ffc000c840d1b0f8d22e592b556b [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:03:49.126385 8935 raft_consensus.cc:3037] T 39da7d3ad5cd4134acd4ff11257b5487 P 529e37385608459bb7d952862fa9e191 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:03:49.129419 8611 raft_consensus.cc:2454] T 39da7d3ad5cd4134acd4ff11257b5487 P a6e3ffc000c840d1b0f8d22e592b556b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b873e04286174b76b4af16a01aea7cbb in term 1.
I20260812 07:03:49.129413 8935 raft_consensus.cc:2454] T 39da7d3ad5cd4134acd4ff11257b5487 P 529e37385608459bb7d952862fa9e191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b873e04286174b76b4af16a01aea7cbb in term 1.
I20260812 07:03:49.129850 8693 leader_election.cc:302] T 39da7d3ad5cd4134acd4ff11257b5487 P b873e04286174b76b4af16a01aea7cbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e3ffc000c840d1b0f8d22e592b556b, b873e04286174b76b4af16a01aea7cbb; no voters:
I20260812 07:03:49.130118 9093 raft_consensus.cc:2781] T 39da7d3ad5cd4134acd4ff11257b5487 P b873e04286174b76b4af16a01aea7cbb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:03:49.130308 9093 raft_consensus.cc:686] T 39da7d3ad5cd4134acd4ff11257b5487 P b873e04286174b76b4af16a01aea7cbb [term 1 LEADER]: Becoming Leader. State: Replica: b873e04286174b76b4af16a01aea7cbb, State: Running, Role: LEADER
I20260812 07:03:49.130600 9093 consensus_queue.cc:237] T 39da7d3ad5cd4134acd4ff11257b5487 P b873e04286174b76b4af16a01aea7cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } }
I20260812 07:03:49.133704 8426 catalog_manager.cc:5495] T 39da7d3ad5cd4134acd4ff11257b5487 P b873e04286174b76b4af16a01aea7cbb reported cstate change: term changed from 0 to 1, leader changed from <none> to b873e04286174b76b4af16a01aea7cbb (127.6.61.2). New cstate: current_term: 1 leader_uuid: "b873e04286174b76b4af16a01aea7cbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b873e04286174b76b4af16a01aea7cbb" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 } health_report { overall_health: UNKNOWN } } }
I20260812 07:03:49.210695 8935 raft_consensus.cc:1261] T 39da7d3ad5cd4134acd4ff11257b5487 P 529e37385608459bb7d952862fa9e191 [term 1 FOLLOWER]: Refusing update from remote peer b873e04286174b76b4af16a01aea7cbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:03:49.210955 8609 raft_consensus.cc:1261] T 39da7d3ad5cd4134acd4ff11257b5487 P a6e3ffc000c840d1b0f8d22e592b556b [term 1 FOLLOWER]: Refusing update from remote peer b873e04286174b76b4af16a01aea7cbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:03:49.211828 9093 consensus_queue.cc:1035] T 39da7d3ad5cd4134acd4ff11257b5487 P b873e04286174b76b4af16a01aea7cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "529e37385608459bb7d952862fa9e191" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:03:49.212255 9098 consensus_queue.cc:1035] T 39da7d3ad5cd4134acd4ff11257b5487 P b873e04286174b76b4af16a01aea7cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e3ffc000c840d1b0f8d22e592b556b" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 07:03:49.242895 9091 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 07:03:50.828004 8426 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:03:50.828284 8426 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:03:50.832854 8426 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 100f02a2c2c24a9f9f301631ca439064: Sending DeleteTablet for 3 replicas of tablet 39da7d3ad5cd4134acd4ff11257b5487
I20260812 07:03:50.834110 8752 tablet_service.cc:1514] Processing DeleteTablet for tablet 39da7d3ad5cd4134acd4ff11257b5487 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:03:50 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:46138
I20260812 07:03:50.834621 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 8511
I20260812 07:03:50.834832 9172 tablet_replica.cc:330] T 39da7d3ad5cd4134acd4ff11257b5487 P b873e04286174b76b4af16a01aea7cbb: stopping tablet replica
I20260812 07:03:50.835094 9172 raft_consensus.cc:2229] T 39da7d3ad5cd4134acd4ff11257b5487 P b873e04286174b76b4af16a01aea7cbb [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:03:50.835525 8585 tablet_service.cc:1514] Processing DeleteTablet for tablet 39da7d3ad5cd4134acd4ff11257b5487 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:03:50 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:37468
I20260812 07:03:50.835649 9172 raft_consensus.cc:2258] T 39da7d3ad5cd4134acd4ff11257b5487 P b873e04286174b76b4af16a01aea7cbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:03:50.836418 8913 tablet_service.cc:1514] Processing DeleteTablet for tablet 39da7d3ad5cd4134acd4ff11257b5487 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:03:50 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:43640
I20260812 07:03:50.842382 9177 tablet_replica.cc:330] T 39da7d3ad5cd4134acd4ff11257b5487 P 529e37385608459bb7d952862fa9e191: stopping tablet replica
I20260812 07:03:50.843046 9174 tablet_replica.cc:330] T 39da7d3ad5cd4134acd4ff11257b5487 P a6e3ffc000c840d1b0f8d22e592b556b: stopping tablet replica
I20260812 07:03:50.843267 9172 ts_tablet_manager.cc:1905] T 39da7d3ad5cd4134acd4ff11257b5487 P b873e04286174b76b4af16a01aea7cbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:03:50.843791 9174 raft_consensus.cc:2229] T 39da7d3ad5cd4134acd4ff11257b5487 P a6e3ffc000c840d1b0f8d22e592b556b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:03:50.843547 9177 raft_consensus.cc:2229] T 39da7d3ad5cd4134acd4ff11257b5487 P 529e37385608459bb7d952862fa9e191 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:03:50.844655 9174 raft_consensus.cc:2258] T 39da7d3ad5cd4134acd4ff11257b5487 P a6e3ffc000c840d1b0f8d22e592b556b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:03:50.844362 9177 raft_consensus.cc:2258] T 39da7d3ad5cd4134acd4ff11257b5487 P 529e37385608459bb7d952862fa9e191 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:03:50.848029 9177 ts_tablet_manager.cc:1905] T 39da7d3ad5cd4134acd4ff11257b5487 P 529e37385608459bb7d952862fa9e191: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:03:50.848068 9174 ts_tablet_manager.cc:1905] T 39da7d3ad5cd4134acd4ff11257b5487 P a6e3ffc000c840d1b0f8d22e592b556b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:03:50.852340 9172 ts_tablet_manager.cc:1918] T 39da7d3ad5cd4134acd4ff11257b5487 P b873e04286174b76b4af16a01aea7cbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 07:03:50.852600 9172 log.cc:1196] T 39da7d3ad5cd4134acd4ff11257b5487 P b873e04286174b76b4af16a01aea7cbb: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-1/wal/wals/39da7d3ad5cd4134acd4ff11257b5487
I20260812 07:03:50.853277 9172 ts_tablet_manager.cc:1939] T 39da7d3ad5cd4134acd4ff11257b5487 P b873e04286174b76b4af16a01aea7cbb: Deleting consensus metadata
I20260812 07:03:50.854674 8410 catalog_manager.cc:4841] TS b873e04286174b76b4af16a01aea7cbb (127.6.61.2:45959): tablet 39da7d3ad5cd4134acd4ff11257b5487 (table test-table [id=fc44146d04104b449bb28e22534064bf]) successfully deleted
I20260812 07:03:50.855129 9174 ts_tablet_manager.cc:1918] T 39da7d3ad5cd4134acd4ff11257b5487 P a6e3ffc000c840d1b0f8d22e592b556b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 07:03:50.855114 9177 ts_tablet_manager.cc:1918] T 39da7d3ad5cd4134acd4ff11257b5487 P 529e37385608459bb7d952862fa9e191: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 07:03:50.855269 9174 log.cc:1196] T 39da7d3ad5cd4134acd4ff11257b5487 P a6e3ffc000c840d1b0f8d22e592b556b: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-0/wal/wals/39da7d3ad5cd4134acd4ff11257b5487
I20260812 07:03:50.855263 9177 log.cc:1196] T 39da7d3ad5cd4134acd4ff11257b5487 P 529e37385608459bb7d952862fa9e191: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518208686283-6388-0/minicluster-data/ts-2/wal/wals/39da7d3ad5cd4134acd4ff11257b5487
I20260812 07:03:50.855773 9177 ts_tablet_manager.cc:1939] T 39da7d3ad5cd4134acd4ff11257b5487 P 529e37385608459bb7d952862fa9e191: Deleting consensus metadata
I20260812 07:03:50.855773 9174 ts_tablet_manager.cc:1939] T 39da7d3ad5cd4134acd4ff11257b5487 P a6e3ffc000c840d1b0f8d22e592b556b: Deleting consensus metadata
I20260812 07:03:50.856899 8408 catalog_manager.cc:4841] TS a6e3ffc000c840d1b0f8d22e592b556b (127.6.61.1:39565): tablet 39da7d3ad5cd4134acd4ff11257b5487 (table test-table [id=fc44146d04104b449bb28e22534064bf]) successfully deleted
I20260812 07:03:50.857122 8412 catalog_manager.cc:4841] TS 529e37385608459bb7d952862fa9e191 (127.6.61.3:38361): tablet 39da7d3ad5cd4134acd4ff11257b5487 (table test-table [id=fc44146d04104b449bb28e22534064bf]) successfully deleted
Aug 12 07:03:50 dist-test-slave-69rg krb5kdc[8377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518228, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.61.1@KRBTEST.COM
I20260812 07:03:50.869706 9175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:50.835966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38302 (local address 127.6.61.1:39565)
0812 07:03:50.846924 (+ 10958us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:50.846933 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:50.847579 (+ 646us) server_negotiation.cc:407] Connection header received
0812 07:03:50.847792 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:50.847799 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:50.847891 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:50.848024 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:50.850314 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:50.851744 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:50.854013 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:50.855140 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:50.855285 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:50.864086 (+ 8801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:50.864104 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:50.864123 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:50.864163 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:50.867125 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:50.867800 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:50.867807 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:50.867810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:50.867949 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:50.868349 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:50.868352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:50.868354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:50.868695 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:50.868911 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:50.869246 (+ 335us) server_negotiation.cc:299] Negotiation successful
0812 07:03:50.869444 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10787,"thread_start_us":10966,"threads_started":1}
W20260812 07:03:50.870569 8662 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38302
W20260812 07:03:50.871310 6388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 8511: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 07:03:50.871564 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 8511
I20260812 07:03:50.901525 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 8673
Aug 12 07:03:50 dist-test-slave-69rg krb5kdc[8377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518228, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.61.2@KRBTEST.COM
I20260812 07:03:50.917784 9178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:50.902559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36390 (local address 127.6.61.2:45959)
0812 07:03:50.903098 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:50.903103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:50.903121 (+ 18us) server_negotiation.cc:407] Connection header received
0812 07:03:50.903186 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:50.903189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:50.903251 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:50.903347 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:50.904749 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:50.905647 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:50.906973 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:50.907389 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:50.907487 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:50.911883 (+ 4396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:50.911902 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:50.911921 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:50.911964 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:50.915199 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:50.915863 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:50.915870 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:50.915872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:50.916006 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:50.916333 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:50.916336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:50.916338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:50.916745 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:50.916978 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:50.917196 (+ 218us) server_negotiation.cc:299] Negotiation successful
0812 07:03:50.917381 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":287,"threads_started":1}
W20260812 07:03:50.918689 8829 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36390
W20260812 07:03:50.919325 6388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 8673: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 07:03:50.919453 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 8673
I20260812 07:03:50.951579 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 8838
Aug 12 07:03:50 dist-test-slave-69rg krb5kdc[8377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518228, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.61.3@KRBTEST.COM
I20260812 07:03:50.967372 9179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:50.952420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42616 (local address 127.6.61.3:38361)
0812 07:03:50.953039 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:50.953044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:50.953058 (+ 14us) server_negotiation.cc:407] Connection header received
0812 07:03:50.953120 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:50.953123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:50.953193 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:50.953319 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:50.954740 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:50.955460 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:50.956690 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:50.957102 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:50.957254 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:50.961687 (+ 4433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:50.961703 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:50.961722 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:50.961755 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:50.964540 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:50.965194 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:50.965203 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:50.965205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:50.965354 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:50.965741 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:50.965746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:50.965749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:50.966164 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:50.966447 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:50.966782 (+ 335us) server_negotiation.cc:299] Negotiation successful
0812 07:03:50.967035 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":345,"threads_started":1}
W20260812 07:03:50.968083 8988 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42616
W20260812 07:03:50.968767 6388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 8838: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 07:03:50.968918 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 8838
I20260812 07:03:50.991767 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 8392
I20260812 07:03:50.993016 8475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:03:51.111822 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 8392
2026-08-12T07:03:51Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8857 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:03:51 dist-test-slave-69rg krb5kdc[9183](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:03:51 dist-test-slave-69rg krb5kdc[9183](info): set up 2 sockets
Aug 12 07:03:51 dist-test-slave-69rg krb5kdc[9183](info): commencing operation
krb5kdc: starting...
W20260812 07:03:52.323419 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:03:52 dist-test-slave-69rg krb5kdc[9183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:03:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:03:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
I20260812 07:03:52.555605 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:45375
--webserver_interface=127.6.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:38613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:45375
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:03:52.895390 9198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:03:52.895668 9198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:03:52.895828 9198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:03:52.903223 9198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:03:52.903337 9198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:03:52.903421 9198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:03:52.903528 9198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:03:52.903594 9198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:03:52.913486 9198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:38613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:45375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:45375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:03:52.914827 9198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:03:52.916236 9198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:03:52.929697 9205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:52.929669 9206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:03:52.930125 9198 server_base.cc:1034] running on GCE node
W20260812 07:03:52.929582 9208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:03:52.931249 9198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:03:52.932775 9198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:03:52.934028 9198 hybrid_clock.cc:648] HybridClock initialized: now 1786518232933969 us; error 47 us; skew 500 ppm
I20260812 07:03:52.934448 9198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:03:52 dist-test-slave-69rg krb5kdc[9183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518232, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:03:52.937896 9198 init.cc:377] Logged in from keytab as kudu/127.6.61.62@KRBTEST.COM (short username kudu)
I20260812 07:03:52.939474 9198 webserver.cc:460] Webserver started at http://127.6.61.62:43475/ using document root <none> and password file <none>
I20260812 07:03:52.940145 9198 fs_manager.cc:359] Metadata directory not provided
I20260812 07:03:52.940251 9198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:03:52.940541 9198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:03:52.943619 9198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "78d0a91ea2714f91b6a30255a15da2a6"
format_stamp: "Formatted at 2026-08-12 07:03:52 on dist-test-slave-69rg"
server_key: "f181b9ac6ccfad44966301b58c5ac820"
server_key_iv: "931085b4152153b41fe7e6e08206c780"
server_key_version: "encryptionkey@0"
I20260812 07:03:52.944371 9198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "78d0a91ea2714f91b6a30255a15da2a6"
format_stamp: "Formatted at 2026-08-12 07:03:52 on dist-test-slave-69rg"
server_key: "f181b9ac6ccfad44966301b58c5ac820"
server_key_iv: "931085b4152153b41fe7e6e08206c780"
server_key_version: "encryptionkey@0"
I20260812 07:03:52.949684 9198 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 07:03:52.953178 9216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:52.954679 9198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:03:52.954862 9198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "78d0a91ea2714f91b6a30255a15da2a6"
format_stamp: "Formatted at 2026-08-12 07:03:52 on dist-test-slave-69rg"
server_key: "f181b9ac6ccfad44966301b58c5ac820"
server_key_iv: "931085b4152153b41fe7e6e08206c780"
server_key_version: "encryptionkey@0"
I20260812 07:03:52.955073 9198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:03:52.966313 9198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:03:52.970971 9198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:03:52.971345 9198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:03:52.992779 9198 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:45375
I20260812 07:03:52.992815 9277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:45375 every 8 connection(s)
I20260812 07:03:52.994437 9198 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:03:52.998355 9278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:03:52.999166 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 9198
I20260812 07:03:52.999478 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:03:53.000205 6388 external_mini_cluster.cc:1428] Setting key dbab938646e5876ebc492b9fa670e20a
I20260812 07:03:53.013196 9278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d0a91ea2714f91b6a30255a15da2a6: Bootstrap starting.
Aug 12 07:03:53 dist-test-slave-69rg krb5kdc[9183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:03:53.016542 9278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78d0a91ea2714f91b6a30255a15da2a6: Neither blocks nor log segments found. Creating new log.
I20260812 07:03:53.017831 9278 log.cc:824] T 00000000000000000000000000000000 P 78d0a91ea2714f91b6a30255a15da2a6: Log is configured to *not* fsync() on all Append() calls
I20260812 07:03:53.021173 9278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d0a91ea2714f91b6a30255a15da2a6: No bootstrap required, opened a new log
I20260812 07:03:53.025475 9281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:53.002583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57228 (local address 127.6.61.62:45375)
0812 07:03:53.003465 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:53.003487 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:53.003538 (+ 51us) server_negotiation.cc:407] Connection header received
0812 07:03:53.005270 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:53.005300 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:53.005853 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:53.006308 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:53.007872 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:53.010025 (+ 2153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:53.011495 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:53.012567 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:53.012762 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:53.017377 (+ 4615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:53.017390 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:53.017411 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:53.017469 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:53.021021 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:53.021917 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:53.021932 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:53.021940 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:53.022124 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:53.022561 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:53.022565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:53.022567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:53.023149 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:53.023513 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:53.023992 (+ 479us) server_negotiation.cc:299] Negotiation successful
0812 07:03:53.024441 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":354,"threads_started":1}
I20260812 07:03:53.029381 9278 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78d0a91ea2714f91b6a30255a15da2a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d0a91ea2714f91b6a30255a15da2a6" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 45375 } }
I20260812 07:03:53.029767 9278 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78d0a91ea2714f91b6a30255a15da2a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:03:53.029917 9278 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78d0a91ea2714f91b6a30255a15da2a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78d0a91ea2714f91b6a30255a15da2a6, State: Initialized, Role: FOLLOWER
I20260812 07:03:53.030476 9278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78d0a91ea2714f91b6a30255a15da2a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d0a91ea2714f91b6a30255a15da2a6" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 45375 } }
I20260812 07:03:53.030712 9278 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78d0a91ea2714f91b6a30255a15da2a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:03:53.030860 9278 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78d0a91ea2714f91b6a30255a15da2a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:03:53.031042 9278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78d0a91ea2714f91b6a30255a15da2a6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:03:53.033282 9278 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78d0a91ea2714f91b6a30255a15da2a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d0a91ea2714f91b6a30255a15da2a6" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 45375 } }
I20260812 07:03:53.033824 9278 leader_election.cc:302] T 00000000000000000000000000000000 P 78d0a91ea2714f91b6a30255a15da2a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78d0a91ea2714f91b6a30255a15da2a6; no voters:
I20260812 07:03:53.034622 9278 leader_election.cc:288] T 00000000000000000000000000000000 P 78d0a91ea2714f91b6a30255a15da2a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:03:53.034730 9283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78d0a91ea2714f91b6a30255a15da2a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:03:53.035462 9283 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78d0a91ea2714f91b6a30255a15da2a6 [term 1 LEADER]: Becoming Leader. State: Replica: 78d0a91ea2714f91b6a30255a15da2a6, State: Running, Role: LEADER
I20260812 07:03:53.036021 9283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78d0a91ea2714f91b6a30255a15da2a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d0a91ea2714f91b6a30255a15da2a6" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 45375 } }
I20260812 07:03:53.036886 9278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78d0a91ea2714f91b6a30255a15da2a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:03:53.039072 9284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d0a91ea2714f91b6a30255a15da2a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78d0a91ea2714f91b6a30255a15da2a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d0a91ea2714f91b6a30255a15da2a6" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 45375 } } }
I20260812 07:03:53.039485 9284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d0a91ea2714f91b6a30255a15da2a6 [sys.catalog]: This master's current role is: LEADER
I20260812 07:03:53.039363 9285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d0a91ea2714f91b6a30255a15da2a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78d0a91ea2714f91b6a30255a15da2a6. Latest consensus state: current_term: 1 leader_uuid: "78d0a91ea2714f91b6a30255a15da2a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d0a91ea2714f91b6a30255a15da2a6" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 45375 } } }
I20260812 07:03:53.039744 9285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d0a91ea2714f91b6a30255a15da2a6 [sys.catalog]: This master's current role is: LEADER
I20260812 07:03:53.041703 9291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:03:53.046826 9291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:03:53.055863 9291 catalog_manager.cc:1347] Generated new cluster ID: 5d481ac734ad4017b68c118f3405e97e
I20260812 07:03:53.055977 9291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:03:53.068979 9291 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:03:53.070940 9291 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:03:53.086207 9291 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78d0a91ea2714f91b6a30255a15da2a6: Generated new TSK 0
I20260812 07:03:53.087337 9291 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
I20260812 07:03:53.158987 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.1:0
--local_ip_for_outbound_sockets=127.6.61.1
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:45375
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:38613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:03:53.499794 9307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:03:53.500131 9307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:03:53.500319 9307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:03:53.507730 9307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:03:53.507886 9307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:03:53.508092 9307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.1
I20260812 07:03:53.518103 9307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:38613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:45375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.1
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:03:53.519474 9307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:03:53.520985 9307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:03:53.535331 9313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:53.535230 9316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:53.535245 9314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:03:53.535976 9307 server_base.cc:1034] running on GCE node
I20260812 07:03:53.536650 9307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:03:53.537704 9307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:03:53.538944 9307 hybrid_clock.cc:648] HybridClock initialized: now 1786518233538903 us; error 72 us; skew 500 ppm
I20260812 07:03:53.539284 9307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:03:53 dist-test-slave-69rg krb5kdc[9183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518233, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:03:53.542819 9307 init.cc:377] Logged in from keytab as kudu/127.6.61.1@KRBTEST.COM (short username kudu)
I20260812 07:03:53.544289 9307 webserver.cc:460] Webserver started at http://127.6.61.1:44147/ using document root <none> and password file <none>
I20260812 07:03:53.545040 9307 fs_manager.cc:359] Metadata directory not provided
I20260812 07:03:53.545197 9307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:03:53.545534 9307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:03:53.548532 9307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-0/data/instance:
uuid: "30ce656139db48e0a9db7c551eb785c7"
format_stamp: "Formatted at 2026-08-12 07:03:53 on dist-test-slave-69rg"
server_key: "6e9d18fe4e94636613b8bb1d8e111413"
server_key_iv: "1e16f724fb0a6ec9e80492bde42a6c38"
server_key_version: "encryptionkey@0"
I20260812 07:03:53.549356 9307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance:
uuid: "30ce656139db48e0a9db7c551eb785c7"
format_stamp: "Formatted at 2026-08-12 07:03:53 on dist-test-slave-69rg"
server_key: "6e9d18fe4e94636613b8bb1d8e111413"
server_key_iv: "1e16f724fb0a6ec9e80492bde42a6c38"
server_key_version: "encryptionkey@0"
I20260812 07:03:53.554528 9307 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 07:03:53.557940 9323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:53.559303 9307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 07:03:53.559484 9307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-0/wal
uuid: "30ce656139db48e0a9db7c551eb785c7"
format_stamp: "Formatted at 2026-08-12 07:03:53 on dist-test-slave-69rg"
server_key: "6e9d18fe4e94636613b8bb1d8e111413"
server_key_iv: "1e16f724fb0a6ec9e80492bde42a6c38"
server_key_version: "encryptionkey@0"
I20260812 07:03:53.559733 9307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:03:53.581590 9307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:03:53.585763 9307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:03:53.586088 9307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:03:53.587013 9307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:03:53.588310 9307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:03:53.588388 9307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:53.588476 9307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:03:53.588521 9307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:53.624387 9307 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.1:33059
I20260812 07:03:53.624487 9450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.1:33059 every 8 connection(s)
I20260812 07:03:53.625744 9307 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
I20260812 07:03:53.633738 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 9307
I20260812 07:03:53.634028 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance
I20260812 07:03:53.634886 6388 external_mini_cluster.cc:1428] Setting key 44b732d464be494c39929137a43b3e39
Aug 12 07:03:53 dist-test-slave-69rg krb5kdc[9183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518233, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:03:53.648967 9453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:53.628715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:53899 (local address 127.6.61.62:45375)
0812 07:03:53.629226 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:53.629231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:53.630252 (+ 1021us) server_negotiation.cc:407] Connection header received
0812 07:03:53.632249 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:53.632254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:53.632327 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:53.632445 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:53.634630 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:53.635486 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:53.637461 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:53.637949 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:53.638113 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:53.641915 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:53.641926 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:53.641935 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:53.641985 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:53.645435 (+ 3450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:53.646524 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:53.646529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:53.646532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:53.646663 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:53.647174 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:53.647177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:53.647180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:53.647468 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:53.647584 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:53.648511 (+ 927us) server_negotiation.cc:299] Negotiation successful
0812 07:03:53.648700 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":253,"threads_started":1}
I20260812 07:03:53.650550 9454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:53.629229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:45375 (local address 127.6.61.1:53899)
0812 07:03:53.630077 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:53.630120 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:53.631952 (+ 1832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:53.632663 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:53.632671 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:53.633396 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:03:53.634404 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:53.634440 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:53.635670 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:53.635676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:53.637306 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:53.637321 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:53.638218 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:53.638227 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:53.638489 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:53.639244 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:03:53.639267 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:03:53.641640 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:03:53.645601 (+ 3961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:53.645611 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:53.645630 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:53.646393 (+ 763us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:53.646784 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:53.646792 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:53.646799 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:53.647044 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:53.647602 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:03:53.647613 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:03:53.648080 (+ 467us) client_negotiation.cc:769] Sending connection context
0812 07:03:53.648401 (+ 321us) client_negotiation.cc:240] Negotiation successful
0812 07:03:53.649018 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":311,"threads_started":1}
I20260812 07:03:53.652318 9451 heartbeater.cc:347] Connected to a master server at 127.6.61.62:45375
I20260812 07:03:53.652667 9451 heartbeater.cc:464] Registering TS with master...
I20260812 07:03:53.653503 9451 heartbeater.cc:511] Master 127.6.61.62:45375 requested a full tablet report, sending...
I20260812 07:03:53.655968 9237 ts_manager.cc:194] Registered new tserver with Master: 30ce656139db48e0a9db7c551eb785c7 (127.6.61.1:33059)
I20260812 07:03:53.657584 9237 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.1@KRBTEST.COM'} at 127.6.61.1:53899
WARNING: no policy specified for kudu/127.6.61.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
I20260812 07:03:53.691192 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.2:0
--local_ip_for_outbound_sockets=127.6.61.2
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:45375
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:38613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:03:54.045169 9459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:03:54.045567 9459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:03:54.045759 9459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:03:54.053459 9459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:03:54.053586 9459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:03:54.053808 9459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.2
I20260812 07:03:54.067219 9459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:38613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:45375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.2
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:03:54.068610 9459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:03:54.070015 9459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:03:54.083465 9468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:54.083420 9466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:54.083504 9465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:03:54.084368 9459 server_base.cc:1034] running on GCE node
I20260812 07:03:54.085035 9459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:03:54.086087 9459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:03:54.087373 9459 hybrid_clock.cc:648] HybridClock initialized: now 1786518234087251 us; error 102 us; skew 500 ppm
I20260812 07:03:54.087793 9459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:03:54 dist-test-slave-69rg krb5kdc[9183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518234, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:03:54.091298 9459 init.cc:377] Logged in from keytab as kudu/127.6.61.2@KRBTEST.COM (short username kudu)
I20260812 07:03:54.092927 9459 webserver.cc:460] Webserver started at http://127.6.61.2:37649/ using document root <none> and password file <none>
I20260812 07:03:54.093616 9459 fs_manager.cc:359] Metadata directory not provided
I20260812 07:03:54.093732 9459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:03:54.094103 9459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:03:54.097288 9459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-1/data/instance:
uuid: "a918b352aacc4e9188c00502313e0a6e"
format_stamp: "Formatted at 2026-08-12 07:03:54 on dist-test-slave-69rg"
server_key: "3904374e83ecc144264e585270072069"
server_key_iv: "2378e4cd48c1f9505e36bf87b60688e2"
server_key_version: "encryptionkey@0"
I20260812 07:03:54.098198 9459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance:
uuid: "a918b352aacc4e9188c00502313e0a6e"
format_stamp: "Formatted at 2026-08-12 07:03:54 on dist-test-slave-69rg"
server_key: "3904374e83ecc144264e585270072069"
server_key_iv: "2378e4cd48c1f9505e36bf87b60688e2"
server_key_version: "encryptionkey@0"
I20260812 07:03:54.103621 9459 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 07:03:54.107149 9475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:54.108479 9459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:03:54.108640 9459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-1/wal
uuid: "a918b352aacc4e9188c00502313e0a6e"
format_stamp: "Formatted at 2026-08-12 07:03:54 on dist-test-slave-69rg"
server_key: "3904374e83ecc144264e585270072069"
server_key_iv: "2378e4cd48c1f9505e36bf87b60688e2"
server_key_version: "encryptionkey@0"
I20260812 07:03:54.108863 9459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:03:54.129773 9459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:03:54.134008 9459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:03:54.134300 9459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:03:54.135270 9459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:03:54.136649 9459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:03:54.136744 9459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:54.136909 9459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:03:54.136962 9459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:54.170971 9459 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.2:46103
I20260812 07:03:54.171015 9605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.2:46103 every 8 connection(s)
I20260812 07:03:54.172271 9459 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
I20260812 07:03:54.178207 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 9459
I20260812 07:03:54.178427 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance
I20260812 07:03:54.178864 6388 external_mini_cluster.cc:1428] Setting key 132e1d64a9c6eb6e0c6472785a2d0a43
Aug 12 07:03:54 dist-test-slave-69rg krb5kdc[9183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518234, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:03:54.192636 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:54.174981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:41707 (local address 127.6.61.62:45375)
0812 07:03:54.175480 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:54.175484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:54.176336 (+ 852us) server_negotiation.cc:407] Connection header received
0812 07:03:54.178227 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:54.178241 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:54.178359 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:54.178474 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:54.180071 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:54.180896 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:54.182462 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:54.182911 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:54.183057 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:54.186556 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:54.186561 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:54.186565 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:54.186596 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:54.189393 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:54.190244 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:54.190249 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:54.190253 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:54.190389 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:54.190850 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:54.190859 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:54.190862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:54.191142 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:54.191307 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:54.192003 (+ 696us) server_negotiation.cc:299] Negotiation successful
0812 07:03:54.192259 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":280,"threads_started":1}
I20260812 07:03:54.193632 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:54.175431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:45375 (local address 127.6.61.2:41707)
0812 07:03:54.176161 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:54.176199 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:54.177928 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:54.178762 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:54.178770 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:54.179336 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:03:54.179886 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:54.179903 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:54.181054 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:54.181060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:54.182304 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:54.182317 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:54.183150 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:54.183159 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:54.183410 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:54.184107 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:03:54.184133 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:03:54.186321 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:03:54.189552 (+ 3231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:54.189560 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:54.189577 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:54.190101 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:54.190511 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:54.190516 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:54.190520 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:54.190719 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:54.191283 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:03:54.191291 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:03:54.191635 (+ 344us) client_negotiation.cc:769] Sending connection context
0812 07:03:54.191909 (+ 274us) client_negotiation.cc:240] Negotiation successful
0812 07:03:54.192445 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":308,"threads_started":1}
I20260812 07:03:54.195432 9606 heartbeater.cc:347] Connected to a master server at 127.6.61.62:45375
I20260812 07:03:54.195802 9606 heartbeater.cc:464] Registering TS with master...
I20260812 07:03:54.196779 9606 heartbeater.cc:511] Master 127.6.61.62:45375 requested a full tablet report, sending...
I20260812 07:03:54.198675 9237 ts_manager.cc:194] Registered new tserver with Master: a918b352aacc4e9188c00502313e0a6e (127.6.61.2:46103)
I20260812 07:03:54.199959 9237 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.2@KRBTEST.COM'} at 127.6.61.2:41707
WARNING: no policy specified for kudu/127.6.61.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
I20260812 07:03:54.229712 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.3:0
--local_ip_for_outbound_sockets=127.6.61.3
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:45375
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:38613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:03:54.563964 9612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:03:54.564231 9612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:03:54.564402 9612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:03:54.571854 9612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:03:54.571966 9612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:03:54.572077 9612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.3
I20260812 07:03:54.581774 9612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:38613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:45375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.3
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:03:54.583042 9612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:03:54.584321 9612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:03:54.597754 9617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:54.598047 9618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:54.598109 9620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:03:54.599781 9612 server_base.cc:1034] running on GCE node
I20260812 07:03:54.600515 9612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:03:54.601532 9612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:03:54.602794 9612 hybrid_clock.cc:648] HybridClock initialized: now 1786518234602723 us; error 88 us; skew 500 ppm
I20260812 07:03:54.603160 9612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:03:54 dist-test-slave-69rg krb5kdc[9183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518234, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:03:54.606776 9612 init.cc:377] Logged in from keytab as kudu/127.6.61.3@KRBTEST.COM (short username kudu)
I20260812 07:03:54.608438 9612 webserver.cc:460] Webserver started at http://127.6.61.3:44513/ using document root <none> and password file <none>
I20260812 07:03:54.609169 9612 fs_manager.cc:359] Metadata directory not provided
I20260812 07:03:54.609329 9612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:03:54.609675 9612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:03:54.613020 9612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-2/data/instance:
uuid: "1042b8a85150488697dd2f6c12dcc04e"
format_stamp: "Formatted at 2026-08-12 07:03:54 on dist-test-slave-69rg"
server_key: "7a7956ed532c9b4ae1083827c1a41e0a"
server_key_iv: "4d437ab809b17b0058a338b6be5c83bd"
server_key_version: "encryptionkey@0"
I20260812 07:03:54.614009 9612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance:
uuid: "1042b8a85150488697dd2f6c12dcc04e"
format_stamp: "Formatted at 2026-08-12 07:03:54 on dist-test-slave-69rg"
server_key: "7a7956ed532c9b4ae1083827c1a41e0a"
server_key_iv: "4d437ab809b17b0058a338b6be5c83bd"
server_key_version: "encryptionkey@0"
I20260812 07:03:54.619377 9612 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260812 07:03:54.623170 9631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:54.625236 9612 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 07:03:54.625423 9612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-2/wal
uuid: "1042b8a85150488697dd2f6c12dcc04e"
format_stamp: "Formatted at 2026-08-12 07:03:54 on dist-test-slave-69rg"
server_key: "7a7956ed532c9b4ae1083827c1a41e0a"
server_key_iv: "4d437ab809b17b0058a338b6be5c83bd"
server_key_version: "encryptionkey@0"
I20260812 07:03:54.625649 9612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:03:54.651153 9612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:03:54.655400 9612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:03:54.655650 9612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:03:54.656571 9612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:03:54.657886 9612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:03:54.657959 9612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:54.658042 9612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:03:54.658102 9612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:54.660820 9451 heartbeater.cc:503] Master 127.6.61.62:45375 was elected leader, sending a full tablet report...
I20260812 07:03:54.694449 9612 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.3:39119
I20260812 07:03:54.694557 9763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.3:39119 every 8 connection(s)
I20260812 07:03:54.695848 9612 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
I20260812 07:03:54.702082 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 9612
I20260812 07:03:54.702317 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance
I20260812 07:03:54.702785 6388 external_mini_cluster.cc:1428] Setting key 50537cc77906b160cb22120deb8e3420
Aug 12 07:03:54 dist-test-slave-69rg krb5kdc[9183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518234, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:03:54.717204 9767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:54.698607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:37195 (local address 127.6.61.62:45375)
0812 07:03:54.699063 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:54.699068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:54.700143 (+ 1075us) server_negotiation.cc:407] Connection header received
0812 07:03:54.702169 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:54.702175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:54.702272 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:54.702401 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:54.704426 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:54.705362 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:54.707179 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:54.707599 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:54.707735 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:54.711314 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:54.711320 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:54.711323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:54.711356 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:54.714074 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:54.714826 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:54.714832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:54.714834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:54.714953 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:54.715416 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:54.715420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:54.715422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:54.715690 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:54.715850 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:54.716564 (+ 714us) server_negotiation.cc:299] Negotiation successful
0812 07:03:54.716795 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":222,"threads_started":1}
I20260812 07:03:54.718295 9768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:54.699154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:45375 (local address 127.6.61.3:37195)
0812 07:03:54.699949 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:54.699999 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:54.701798 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:54.702662 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:54.702671 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:54.703330 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:03:54.704254 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:54.704282 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:54.705521 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:54.705528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:54.707007 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:54.707021 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:54.707844 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:54.707852 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:54.708082 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:54.708732 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:03:54.708755 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:03:54.711097 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:03:54.714218 (+ 3121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:54.714227 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:54.714244 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:54.714681 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:54.715072 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:54.715076 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:54.715080 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:54.715269 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:54.715827 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:03:54.715835 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:03:54.716175 (+ 340us) client_negotiation.cc:769] Sending connection context
0812 07:03:54.716454 (+ 279us) client_negotiation.cc:240] Negotiation successful
0812 07:03:54.716996 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":309,"threads_started":1}
I20260812 07:03:54.719695 9764 heartbeater.cc:347] Connected to a master server at 127.6.61.62:45375
I20260812 07:03:54.719974 9764 heartbeater.cc:464] Registering TS with master...
I20260812 07:03:54.720664 9764 heartbeater.cc:511] Master 127.6.61.62:45375 requested a full tablet report, sending...
I20260812 07:03:54.722090 9237 ts_manager.cc:194] Registered new tserver with Master: 1042b8a85150488697dd2f6c12dcc04e (127.6.61.3:39119)
I20260812 07:03:54.722893 9237 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.3@KRBTEST.COM'} at 127.6.61.3:37195
I20260812 07:03:54.731012 6388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 07:03:54.767882 9767 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:03:54.765076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57242 (local address 127.6.61.62:45375)
0812 07:03:54.765373 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:54.765381 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:54.765791 (+ 410us) server_negotiation.cc:407] Connection header received
0812 07:03:54.766051 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:54.766056 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:54.766127 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:54.766243 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:54.767585 (+ 1342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57242: BlockingRecv error: recv got EOF from 127.0.0.1:57242 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
I20260812 07:03:54.771472 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 9307
W20260812 07:03:54.774473 6388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 9307: Not authorized: Client connection negotiation failed: client connection to 127.6.61.1:33059: server requires authentication, but client does not have Kerberos credentials available
I20260812 07:03:54.774698 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 9307
I20260812 07:03:54.795778 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 9459
W20260812 07:03:54.798620 6388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 9459: Not authorized: Client connection negotiation failed: client connection to 127.6.61.2:46103: server requires authentication, but client does not have Kerberos credentials available
I20260812 07:03:54.798815 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 9459
I20260812 07:03:54.818037 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 9612
W20260812 07:03:54.821051 6388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 9612: Not authorized: Client connection negotiation failed: client connection to 127.6.61.3:39119: server requires authentication, but client does not have Kerberos credentials available
I20260812 07:03:54.821209 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 9612
I20260812 07:03:54.839921 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 9198
I20260812 07:03:54.841225 9272 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:03:54.943647 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 9198
2026-08-12T07:03:54Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3835 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:03:55 dist-test-slave-69rg krb5kdc[9789](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:03:55 dist-test-slave-69rg krb5kdc[9789](info): set up 2 sockets
Aug 12 07:03:55 dist-test-slave-69rg krb5kdc[9789](info): commencing operation
krb5kdc: starting...
W20260812 07:03:56.198583 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.202s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:03:56 dist-test-slave-69rg krb5kdc[9789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:03:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:03:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
I20260812 07:03:56.429286 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:40755
--webserver_interface=127.6.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:35205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:40755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:03:56.766196 9802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:03:56.766497 9802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:03:56.766588 9802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:03:56.773880 9802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:03:56.773994 9802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:03:56.774040 9802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:03:56.774091 9802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:03:56.774132 9802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:03:56.784775 9802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:35205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:40755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:40755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:03:56.786114 9802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:03:56.787393 9802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:03:56.799609 9808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:56.799548 9809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:03:56.800168 9802 server_base.cc:1034] running on GCE node
W20260812 07:03:56.800240 9811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:03:56.801064 9802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:03:56.802565 9802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:03:56.803829 9802 hybrid_clock.cc:648] HybridClock initialized: now 1786518236803797 us; error 50 us; skew 500 ppm
I20260812 07:03:56.804260 9802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:03:56 dist-test-slave-69rg krb5kdc[9789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518236, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:03:56.813093 9802 init.cc:377] Logged in from keytab as kudu/127.6.61.62@KRBTEST.COM (short username kudu)
I20260812 07:03:56.814843 9802 webserver.cc:460] Webserver started at http://127.6.61.62:35273/ using document root <none> and password file <none>
I20260812 07:03:56.815579 9802 fs_manager.cc:359] Metadata directory not provided
I20260812 07:03:56.815743 9802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:03:56.816075 9802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:03:56.819324 9802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "fa43dcac216b4b05a2aa83158a93a512"
format_stamp: "Formatted at 2026-08-12 07:03:56 on dist-test-slave-69rg"
server_key: "2aa12f1755b693d181fc73c05e9f4dff"
server_key_iv: "fb7fd5d8dc602cc5c1d6c00141a51e21"
server_key_version: "encryptionkey@0"
I20260812 07:03:56.820153 9802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "fa43dcac216b4b05a2aa83158a93a512"
format_stamp: "Formatted at 2026-08-12 07:03:56 on dist-test-slave-69rg"
server_key: "2aa12f1755b693d181fc73c05e9f4dff"
server_key_iv: "fb7fd5d8dc602cc5c1d6c00141a51e21"
server_key_version: "encryptionkey@0"
I20260812 07:03:56.825495 9802 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 07:03:56.829306 9821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:56.830659 9802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:03:56.830888 9802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "fa43dcac216b4b05a2aa83158a93a512"
format_stamp: "Formatted at 2026-08-12 07:03:56 on dist-test-slave-69rg"
server_key: "2aa12f1755b693d181fc73c05e9f4dff"
server_key_iv: "fb7fd5d8dc602cc5c1d6c00141a51e21"
server_key_version: "encryptionkey@0"
I20260812 07:03:56.831108 9802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:03:56.845170 9802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:03:56.849402 9802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:03:56.849712 9802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:03:56.869495 9802 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:40755
I20260812 07:03:56.869539 9879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:40755 every 8 connection(s)
I20260812 07:03:56.870965 9802 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:03:56.871925 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 9802
I20260812 07:03:56.872226 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:03:56.872964 6388 external_mini_cluster.cc:1428] Setting key 008b053d7f9cb9fbabd659ea74b567d5
I20260812 07:03:56.875837 9880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 07:03:56 dist-test-slave-69rg krb5kdc[9789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:03:56.889415 9880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa43dcac216b4b05a2aa83158a93a512: Bootstrap starting.
I20260812 07:03:56.892724 9880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa43dcac216b4b05a2aa83158a93a512: Neither blocks nor log segments found. Creating new log.
I20260812 07:03:56.893766 9880 log.cc:824] T 00000000000000000000000000000000 P fa43dcac216b4b05a2aa83158a93a512: Log is configured to *not* fsync() on all Append() calls
I20260812 07:03:56.896732 9880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa43dcac216b4b05a2aa83158a93a512: No bootstrap required, opened a new log
I20260812 07:03:56.897142 9883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:56.874967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54752 (local address 127.6.61.62:40755)
0812 07:03:56.875848 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:56.875860 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:56.875903 (+ 43us) server_negotiation.cc:407] Connection header received
0812 07:03:56.877569 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:56.877598 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:56.878048 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:56.878403 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:56.880384 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:56.881912 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:56.883706 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:56.884507 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:56.884667 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:56.889623 (+ 4956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:56.889639 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:56.889655 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:56.889694 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:56.892782 (+ 3088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:56.893558 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:56.893571 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:56.893578 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:56.893765 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:56.894270 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:56.894276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:56.894279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:56.894861 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:56.895176 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:56.895665 (+ 489us) server_negotiation.cc:299] Negotiation successful
0812 07:03:56.896106 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":427,"threads_started":1}
I20260812 07:03:56.901417 6388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:03:56.905388 9880 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa43dcac216b4b05a2aa83158a93a512 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa43dcac216b4b05a2aa83158a93a512" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 40755 } }
I20260812 07:03:56.905781 9880 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa43dcac216b4b05a2aa83158a93a512 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:03:56.905977 9880 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa43dcac216b4b05a2aa83158a93a512 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa43dcac216b4b05a2aa83158a93a512, State: Initialized, Role: FOLLOWER
I20260812 07:03:56.906535 9880 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa43dcac216b4b05a2aa83158a93a512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa43dcac216b4b05a2aa83158a93a512" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 40755 } }
I20260812 07:03:56.906783 9880 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa43dcac216b4b05a2aa83158a93a512 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:03:56.906936 9880 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa43dcac216b4b05a2aa83158a93a512 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:03:56.907132 9880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa43dcac216b4b05a2aa83158a93a512 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:03:56.909405 9880 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa43dcac216b4b05a2aa83158a93a512 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa43dcac216b4b05a2aa83158a93a512" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 40755 } }
I20260812 07:03:56.909986 9880 leader_election.cc:302] T 00000000000000000000000000000000 P fa43dcac216b4b05a2aa83158a93a512 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa43dcac216b4b05a2aa83158a93a512; no voters:
I20260812 07:03:56.910696 9880 leader_election.cc:288] T 00000000000000000000000000000000 P fa43dcac216b4b05a2aa83158a93a512 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:03:56.910815 9887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa43dcac216b4b05a2aa83158a93a512 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:03:56.912312 9887 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa43dcac216b4b05a2aa83158a93a512 [term 1 LEADER]: Becoming Leader. State: Replica: fa43dcac216b4b05a2aa83158a93a512, State: Running, Role: LEADER
I20260812 07:03:56.912874 9887 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa43dcac216b4b05a2aa83158a93a512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa43dcac216b4b05a2aa83158a93a512" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 40755 } }
I20260812 07:03:56.913195 9880 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa43dcac216b4b05a2aa83158a93a512 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:03:56.915963 9888 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa43dcac216b4b05a2aa83158a93a512 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa43dcac216b4b05a2aa83158a93a512" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa43dcac216b4b05a2aa83158a93a512" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 40755 } } }
I20260812 07:03:56.916383 9888 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa43dcac216b4b05a2aa83158a93a512 [sys.catalog]: This master's current role is: LEADER
I20260812 07:03:56.918004 9890 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa43dcac216b4b05a2aa83158a93a512 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa43dcac216b4b05a2aa83158a93a512. Latest consensus state: current_term: 1 leader_uuid: "fa43dcac216b4b05a2aa83158a93a512" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa43dcac216b4b05a2aa83158a93a512" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 40755 } } }
I20260812 07:03:56.918462 9890 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa43dcac216b4b05a2aa83158a93a512 [sys.catalog]: This master's current role is: LEADER
I20260812 07:03:56.922611 9893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:03:56.929054 9893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:03:56.940400 9893 catalog_manager.cc:1347] Generated new cluster ID: 00385ff3e64945419419173ab96f08af
I20260812 07:03:56.940539 9893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:03:56.960199 9883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:56.950218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54768 (local address 127.6.61.62:40755)
0812 07:03:56.950407 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:56.950413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:56.951073 (+ 660us) server_negotiation.cc:407] Connection header received
0812 07:03:56.951279 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:56.951286 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:56.951372 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:56.951492 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:56.952822 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:56.954141 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:56.956047 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:56.956771 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:56.956911 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:56.958268 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:56.958276 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:56.958281 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:03:56.959224 (+ 943us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:56.959284 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:03:56.959326 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:56.959442 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:56.959627 (+ 185us) server_negotiation.cc:299] Negotiation successful
0812 07:03:56.959739 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 07:03:56.964462 9893 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:03:56.965759 9893 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:03:56.980093 9893 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa43dcac216b4b05a2aa83158a93a512: Generated new TSK 0
I20260812 07:03:56.981051 9893 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:03:56.995821 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 9802
I20260812 07:03:56.997731 9875 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:03:57.091135 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 9802
2026-08-12T07:03:57Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2136 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:03:57 dist-test-slave-69rg krb5kdc[9918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:03:57 dist-test-slave-69rg krb5kdc[9918](info): set up 2 sockets
Aug 12 07:03:57 dist-test-slave-69rg krb5kdc[9918](info): commencing operation
krb5kdc: starting...
W20260812 07:03:58.265250 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.008s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:03:58 dist-test-slave-69rg krb5kdc[9918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:03:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:03:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
I20260812 07:03:58.494433 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:33529
--webserver_interface=127.6.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:40933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:33529
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:03:58.828251 9937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:03:58.828611 9937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:03:58.828778 9937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:03:58.836337 9937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:03:58.836488 9937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:03:58.836596 9937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:03:58.836700 9937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:03:58.836789 9937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:03:58.846916 9937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:40933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:33529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:33529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:03:58.848310 9937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:03:58.849647 9937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:03:58.862121 9943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:58.862072 9944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:58.862157 9947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:03:58.863121 9937 server_base.cc:1034] running on GCE node
I20260812 07:03:58.863991 9937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:03:58.865422 9937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:03:58.866660 9937 hybrid_clock.cc:648] HybridClock initialized: now 1786518238866625 us; error 60 us; skew 500 ppm
I20260812 07:03:58.867013 9937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:03:58 dist-test-slave-69rg krb5kdc[9918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518238, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:03:58.870263 9937 init.cc:377] Logged in from keytab as kudu/127.6.61.62@KRBTEST.COM (short username kudu)
I20260812 07:03:58.871737 9937 webserver.cc:460] Webserver started at http://127.6.61.62:45099/ using document root <none> and password file <none>
I20260812 07:03:58.872439 9937 fs_manager.cc:359] Metadata directory not provided
I20260812 07:03:58.872603 9937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:03:58.872924 9937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:03:58.875993 9937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "2babbc369e834fd7ae6c051bef533064"
format_stamp: "Formatted at 2026-08-12 07:03:58 on dist-test-slave-69rg"
server_key: "db5ecf297d6353a39495f33b5b691a2b"
server_key_iv: "5c76014a09cedfe3f600f831259ac78a"
server_key_version: "encryptionkey@0"
I20260812 07:03:58.876780 9937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "2babbc369e834fd7ae6c051bef533064"
format_stamp: "Formatted at 2026-08-12 07:03:58 on dist-test-slave-69rg"
server_key: "db5ecf297d6353a39495f33b5b691a2b"
server_key_iv: "5c76014a09cedfe3f600f831259ac78a"
server_key_version: "encryptionkey@0"
I20260812 07:03:58.882244 9937 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260812 07:03:58.885733 9955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:58.887226 9937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:03:58.887445 9937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "2babbc369e834fd7ae6c051bef533064"
format_stamp: "Formatted at 2026-08-12 07:03:58 on dist-test-slave-69rg"
server_key: "db5ecf297d6353a39495f33b5b691a2b"
server_key_iv: "5c76014a09cedfe3f600f831259ac78a"
server_key_version: "encryptionkey@0"
I20260812 07:03:58.887665 9937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:03:58.912556 9937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:03:58.917245 9937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:03:58.917574 9937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:03:58.937321 9937 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:33529
I20260812 07:03:58.937357 10014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:33529 every 8 connection(s)
I20260812 07:03:58.938753 9937 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:03:58.942387 10015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:03:58.946677 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 9937
I20260812 07:03:58.946899 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:03:58.947741 6388 external_mini_cluster.cc:1428] Setting key f174e50357497989bebfd91171433001
I20260812 07:03:58.954572 10015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2babbc369e834fd7ae6c051bef533064: Bootstrap starting.
I20260812 07:03:58.958230 10015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2babbc369e834fd7ae6c051bef533064: Neither blocks nor log segments found. Creating new log.
I20260812 07:03:58.959370 10015 log.cc:824] T 00000000000000000000000000000000 P 2babbc369e834fd7ae6c051bef533064: Log is configured to *not* fsync() on all Append() calls
I20260812 07:03:58.962950 10015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2babbc369e834fd7ae6c051bef533064: No bootstrap required, opened a new log
Aug 12 07:03:58 dist-test-slave-69rg krb5kdc[9918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:03:58.973750 10019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:58.949914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58162 (local address 127.6.61.62:33529)
0812 07:03:58.950687 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:58.950698 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:58.950740 (+ 42us) server_negotiation.cc:407] Connection header received
0812 07:03:58.953003 (+ 2263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:58.953025 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:58.953416 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:58.953720 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:58.955573 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:58.956875 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:58.958738 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:58.959404 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:58.959604 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:58.964600 (+ 4996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:58.964611 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:58.964636 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:58.964688 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:58.968882 (+ 4194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:58.969699 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:58.969707 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:58.969717 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:58.969957 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:58.970337 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:58.970342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:58.970345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:58.971098 (+ 753us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:58.971429 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:58.971788 (+ 359us) server_negotiation.cc:299] Negotiation successful
0812 07:03:58.972295 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":370,"threads_started":1}
I20260812 07:03:58.974146 10015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2babbc369e834fd7ae6c051bef533064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2babbc369e834fd7ae6c051bef533064" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 33529 } }
I20260812 07:03:58.974547 10015 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2babbc369e834fd7ae6c051bef533064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:03:58.974681 10015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2babbc369e834fd7ae6c051bef533064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2babbc369e834fd7ae6c051bef533064, State: Initialized, Role: FOLLOWER
I20260812 07:03:58.975261 10015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2babbc369e834fd7ae6c051bef533064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2babbc369e834fd7ae6c051bef533064" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 33529 } }
I20260812 07:03:58.975551 10015 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2babbc369e834fd7ae6c051bef533064 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:03:58.975721 10015 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2babbc369e834fd7ae6c051bef533064 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:03:58.975915 10015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2babbc369e834fd7ae6c051bef533064 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:03:58.978452 10015 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2babbc369e834fd7ae6c051bef533064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2babbc369e834fd7ae6c051bef533064" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 33529 } }
I20260812 07:03:58.979010 10015 leader_election.cc:302] T 00000000000000000000000000000000 P 2babbc369e834fd7ae6c051bef533064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2babbc369e834fd7ae6c051bef533064; no voters:
I20260812 07:03:58.979610 10015 leader_election.cc:288] T 00000000000000000000000000000000 P 2babbc369e834fd7ae6c051bef533064 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:03:58.979749 10021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2babbc369e834fd7ae6c051bef533064 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:03:58.980542 10021 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2babbc369e834fd7ae6c051bef533064 [term 1 LEADER]: Becoming Leader. State: Replica: 2babbc369e834fd7ae6c051bef533064, State: Running, Role: LEADER
I20260812 07:03:58.981153 10021 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2babbc369e834fd7ae6c051bef533064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2babbc369e834fd7ae6c051bef533064" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 33529 } }
I20260812 07:03:58.981756 10015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2babbc369e834fd7ae6c051bef533064 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:03:58.984220 10024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2babbc369e834fd7ae6c051bef533064 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2babbc369e834fd7ae6c051bef533064. Latest consensus state: current_term: 1 leader_uuid: "2babbc369e834fd7ae6c051bef533064" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2babbc369e834fd7ae6c051bef533064" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 33529 } } }
I20260812 07:03:58.984660 10024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2babbc369e834fd7ae6c051bef533064 [sys.catalog]: This master's current role is: LEADER
I20260812 07:03:58.985203 10022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2babbc369e834fd7ae6c051bef533064 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2babbc369e834fd7ae6c051bef533064" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2babbc369e834fd7ae6c051bef533064" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 33529 } } }
I20260812 07:03:58.985615 10022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2babbc369e834fd7ae6c051bef533064 [sys.catalog]: This master's current role is: LEADER
I20260812 07:03:58.985646 10033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:03:58.991118 10033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:03:59.000828 10033 catalog_manager.cc:1347] Generated new cluster ID: f8baeec288b64e6b8af4f33c467738e9
I20260812 07:03:59.000976 10033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:03:59.012815 10033 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:03:59.015002 10033 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:03:59.024627 10033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2babbc369e834fd7ae6c051bef533064: Generated new TSK 0
I20260812 07:03:59.025581 10033 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
I20260812 07:03:59.084002 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.1:0
--local_ip_for_outbound_sockets=127.6.61.1
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:33529
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:40933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:03:59.434963 10048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:03:59.435281 10048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:03:59.435465 10048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:03:59.442790 10048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:03:59.442906 10048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:03:59.443022 10048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.1
I20260812 07:03:59.452907 10048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:40933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:33529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.1
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:03:59.454190 10048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:03:59.455459 10048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:03:59.468699 10054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:59.468792 10056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:59.468767 10053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:03:59.469520 10048 server_base.cc:1034] running on GCE node
I20260812 07:03:59.470160 10048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:03:59.471036 10048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:03:59.472234 10048 hybrid_clock.cc:648] HybridClock initialized: now 1786518239472173 us; error 52 us; skew 500 ppm
I20260812 07:03:59.472597 10048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:03:59 dist-test-slave-69rg krb5kdc[9918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518239, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:03:59.476012 10048 init.cc:377] Logged in from keytab as kudu/127.6.61.1@KRBTEST.COM (short username kudu)
I20260812 07:03:59.477603 10048 webserver.cc:460] Webserver started at http://127.6.61.1:37193/ using document root <none> and password file <none>
I20260812 07:03:59.478394 10048 fs_manager.cc:359] Metadata directory not provided
I20260812 07:03:59.478551 10048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:03:59.478876 10048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:03:59.482059 10048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-0/data/instance:
uuid: "48c558775a7841b492aabfbb6ad19e3e"
format_stamp: "Formatted at 2026-08-12 07:03:59 on dist-test-slave-69rg"
server_key: "5d27432a16385624f92a4b7452842caf"
server_key_iv: "12adcf7f14e8736348c80512048b71a2"
server_key_version: "encryptionkey@0"
I20260812 07:03:59.482906 10048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance:
uuid: "48c558775a7841b492aabfbb6ad19e3e"
format_stamp: "Formatted at 2026-08-12 07:03:59 on dist-test-slave-69rg"
server_key: "5d27432a16385624f92a4b7452842caf"
server_key_iv: "12adcf7f14e8736348c80512048b71a2"
server_key_version: "encryptionkey@0"
I20260812 07:03:59.488243 10048 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260812 07:03:59.491797 10066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:59.493079 10048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 07:03:59.493253 10048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-0/wal
uuid: "48c558775a7841b492aabfbb6ad19e3e"
format_stamp: "Formatted at 2026-08-12 07:03:59 on dist-test-slave-69rg"
server_key: "5d27432a16385624f92a4b7452842caf"
server_key_iv: "12adcf7f14e8736348c80512048b71a2"
server_key_version: "encryptionkey@0"
I20260812 07:03:59.493459 10048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:03:59.507752 10048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:03:59.511551 10048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:03:59.511868 10048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:03:59.512795 10048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:03:59.514174 10048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:03:59.514284 10048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:59.514429 10048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:03:59.514504 10048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:03:59.548751 10048 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.1:41157
I20260812 07:03:59.548794 10200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.1:41157 every 8 connection(s)
I20260812 07:03:59.550133 10048 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
I20260812 07:03:59.558120 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 10048
I20260812 07:03:59.558367 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance
I20260812 07:03:59.559026 6388 external_mini_cluster.cc:1428] Setting key 770d69003c127c0ed300615e78ae0685
Aug 12 07:03:59 dist-test-slave-69rg krb5kdc[9918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518239, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:03:59.570577 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:59.552748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:51091 (local address 127.6.61.62:33529)
0812 07:03:59.553203 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0812 07:03:59.553208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:03:59.554247 (+ 1039us) server_negotiation.cc:407] Connection header received
0812 07:03:59.556020 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:03:59.556026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:03:59.556125 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:03:59.556275 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:03:59.557733 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:59.558717 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:59.560468 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:03:59.561014 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:03:59.561158 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:59.564824 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:03:59.564830 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:03:59.564835 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:03:59.564878 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:03:59.567503 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:59.568227 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:59.568242 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:59.568246 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:59.568375 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:03:59.568817 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:03:59.568823 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:03:59.568827 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:03:59.569093 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:03:59.569268 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0812 07:03:59.569970 (+ 702us) server_negotiation.cc:299] Negotiation successful
0812 07:03:59.570197 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":225,"threads_started":1}
I20260812 07:03:59.571501 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:03:59.553320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:33529 (local address 127.6.61.1:51091)
0812 07:03:59.554085 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0812 07:03:59.554121 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 07:03:59.555748 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:03:59.556508 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:03:59.556516 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:03:59.557001 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:03:59.557552 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:59.557569 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:59.558882 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:59.558894 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:59.560308 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:03:59.560321 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:03:59.561241 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:03:59.561249 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:03:59.561508 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:03:59.562219 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:03:59.562244 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:03:59.564602 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:03:59.567650 (+ 3048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:59.567659 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:59.567674 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:59.568099 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:59.568496 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:03:59.568501 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:03:59.568505 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:03:59.568693 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:03:59.569230 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:03:59.569238 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:03:59.569575 (+ 337us) client_negotiation.cc:769] Sending connection context
0812 07:03:59.569866 (+ 291us) client_negotiation.cc:240] Negotiation successful
0812 07:03:59.570385 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":284,"threads_started":1}
I20260812 07:03:59.573093 10201 heartbeater.cc:347] Connected to a master server at 127.6.61.62:33529
I20260812 07:03:59.573412 10201 heartbeater.cc:464] Registering TS with master...
I20260812 07:03:59.574189 10201 heartbeater.cc:511] Master 127.6.61.62:33529 requested a full tablet report, sending...
I20260812 07:03:59.576493 9976 ts_manager.cc:194] Registered new tserver with Master: 48c558775a7841b492aabfbb6ad19e3e (127.6.61.1:41157)
I20260812 07:03:59.578850 9976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.1@KRBTEST.COM'} at 127.6.61.1:51091
WARNING: no policy specified for kudu/127.6.61.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
I20260812 07:03:59.614661 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.2:0
--local_ip_for_outbound_sockets=127.6.61.2
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:33529
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:40933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:03:59.947826 10207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:03:59.948091 10207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:03:59.948261 10207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:03:59.955631 10207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:03:59.955740 10207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:03:59.955914 10207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.2
I20260812 07:03:59.965869 10207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:40933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:33529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.2
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:03:59.967175 10207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:03:59.968477 10207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:03:59.982331 10219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:03:59.983180 10207 server_base.cc:1034] running on GCE node
W20260812 07:03:59.982373 10214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:03:59.982299 10217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:03:59.984643 10207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:03:59.985596 10207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:03:59.986824 10207 hybrid_clock.cc:648] HybridClock initialized: now 1786518239986752 us; error 55 us; skew 500 ppm
I20260812 07:03:59.987193 10207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:03:59 dist-test-slave-69rg krb5kdc[9918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518239, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:03:59.990595 10207 init.cc:377] Logged in from keytab as kudu/127.6.61.2@KRBTEST.COM (short username kudu)
I20260812 07:03:59.992280 10207 webserver.cc:460] Webserver started at http://127.6.61.2:43671/ using document root <none> and password file <none>
I20260812 07:03:59.993076 10207 fs_manager.cc:359] Metadata directory not provided
I20260812 07:03:59.993225 10207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:03:59.993551 10207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:03:59.996793 10207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-1/data/instance:
uuid: "e96c27ae5c9748c6b69df5f770629aac"
format_stamp: "Formatted at 2026-08-12 07:03:59 on dist-test-slave-69rg"
server_key: "9864f9cb1c9538d5e4de2f75f3c05d59"
server_key_iv: "9dede3065b8bc5de12c946ab6f0749e7"
server_key_version: "encryptionkey@0"
I20260812 07:03:59.997596 10207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance:
uuid: "e96c27ae5c9748c6b69df5f770629aac"
format_stamp: "Formatted at 2026-08-12 07:03:59 on dist-test-slave-69rg"
server_key: "9864f9cb1c9538d5e4de2f75f3c05d59"
server_key_iv: "9dede3065b8bc5de12c946ab6f0749e7"
server_key_version: "encryptionkey@0"
I20260812 07:04:00.002954 10207 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 07:04:00.006664 10227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:00.008046 10207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:04:00.008252 10207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-1/wal
uuid: "e96c27ae5c9748c6b69df5f770629aac"
format_stamp: "Formatted at 2026-08-12 07:03:59 on dist-test-slave-69rg"
server_key: "9864f9cb1c9538d5e4de2f75f3c05d59"
server_key_iv: "9dede3065b8bc5de12c946ab6f0749e7"
server_key_version: "encryptionkey@0"
I20260812 07:04:00.008476 10207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:00.027742 10207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:00.032326 10207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:00.032675 10207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:00.033866 10207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:00.035250 10207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:00.035329 10207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:00.035503 10207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:00.035557 10207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:00.072677 10207 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.2:36969
I20260812 07:04:00.072685 10359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.2:36969 every 8 connection(s)
I20260812 07:04:00.074136 10207 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
I20260812 07:04:00.078181 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 10207
I20260812 07:04:00.078368 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance
I20260812 07:04:00.078835 6388 external_mini_cluster.cc:1428] Setting key b24ed3e136bf12ffcef4055fd9ea7773
Aug 12 07:04:00 dist-test-slave-69rg krb5kdc[9918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518239, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:04:00.095677 10362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:00.076801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:39285 (local address 127.6.61.62:33529)
0812 07:04:00.077271 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:00.077276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:00.078213 (+ 937us) server_negotiation.cc:407] Connection header received
0812 07:04:00.080285 (+ 2072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:00.080290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:00.080384 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:00.080509 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:00.082318 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:00.083220 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:00.084850 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:00.085279 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:00.085433 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:00.089285 (+ 3852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:00.089290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:00.089294 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:00.089329 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:00.092331 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:00.093124 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:00.093130 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:00.093133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:00.093286 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:00.093761 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:00.093772 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:00.093775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:00.094119 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:00.094297 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:00.095040 (+ 743us) server_negotiation.cc:299] Negotiation successful
0812 07:04:00.095291 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":253,"threads_started":1}
I20260812 07:04:00.096673 10364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:00.077201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:33529 (local address 127.6.61.2:39285)
0812 07:04:00.078021 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:00.078067 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:00.080015 (+ 1948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:00.080751 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:00.080760 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:00.081440 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:00.082139 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:00.082159 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:00.083390 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:00.083394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:00.084682 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:00.084694 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:00.085550 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:00.085559 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:00.085796 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:00.086661 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:00.086689 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:00.089032 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:00.092493 (+ 3461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:00.092503 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:00.092520 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:00.092974 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:00.093412 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:00.093417 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:00.093420 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:00.093625 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:00.094262 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:00.094270 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:00.094638 (+ 368us) client_negotiation.cc:769] Sending connection context
0812 07:04:00.094928 (+ 290us) client_negotiation.cc:240] Negotiation successful
0812 07:04:00.095489 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":295,"threads_started":1}
I20260812 07:04:00.098305 10360 heartbeater.cc:347] Connected to a master server at 127.6.61.62:33529
I20260812 07:04:00.098645 10360 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:00.099424 10360 heartbeater.cc:511] Master 127.6.61.62:33529 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.61.3@KRBTEST.COM; defaulting to no policy
I20260812 07:04:00.101025 9976 ts_manager.cc:194] Registered new tserver with Master: e96c27ae5c9748c6b69df5f770629aac (127.6.61.2:36969)
I20260812 07:04:00.101929 9976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.2@KRBTEST.COM'} at 127.6.61.2:39285
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
I20260812 07:04:00.154058 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.3:0
--local_ip_for_outbound_sockets=127.6.61.3
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:33529
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:40933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:00.489530 10371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:00.489801 10371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:00.490020 10371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:00.497498 10371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:00.497608 10371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:00.497784 10371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.3
I20260812 07:04:00.507874 10371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:40933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:33529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.3
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:00.509236 10371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:00.510635 10371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:00.524500 10377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:00.524441 10378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:00.524554 10380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:00.525803 10371 server_base.cc:1034] running on GCE node
I20260812 07:04:00.526609 10371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:00.527655 10371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:00.528924 10371 hybrid_clock.cc:648] HybridClock initialized: now 1786518240528851 us; error 54 us; skew 500 ppm
I20260812 07:04:00.529320 10371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:04:00 dist-test-slave-69rg krb5kdc[9918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518240, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:00.532819 10371 init.cc:377] Logged in from keytab as kudu/127.6.61.3@KRBTEST.COM (short username kudu)
I20260812 07:04:00.534641 10371 webserver.cc:460] Webserver started at http://127.6.61.3:40023/ using document root <none> and password file <none>
I20260812 07:04:00.535422 10371 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:00.535596 10371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:00.535933 10371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:00.539238 10371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-2/data/instance:
uuid: "964d7e6fea5e476e9548ee86ec1bb589"
format_stamp: "Formatted at 2026-08-12 07:04:00 on dist-test-slave-69rg"
server_key: "75a5423d5b4132d65c922229f96b4435"
server_key_iv: "01b4e3d10c58faba11a373f1ab8274cc"
server_key_version: "encryptionkey@0"
I20260812 07:04:00.540112 10371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance:
uuid: "964d7e6fea5e476e9548ee86ec1bb589"
format_stamp: "Formatted at 2026-08-12 07:04:00 on dist-test-slave-69rg"
server_key: "75a5423d5b4132d65c922229f96b4435"
server_key_iv: "01b4e3d10c58faba11a373f1ab8274cc"
server_key_version: "encryptionkey@0"
I20260812 07:04:00.545625 10371 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260812 07:04:00.549492 10388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:00.551062 10371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:04:00.551226 10371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-2/wal
uuid: "964d7e6fea5e476e9548ee86ec1bb589"
format_stamp: "Formatted at 2026-08-12 07:04:00 on dist-test-slave-69rg"
server_key: "75a5423d5b4132d65c922229f96b4435"
server_key_iv: "01b4e3d10c58faba11a373f1ab8274cc"
server_key_version: "encryptionkey@0"
I20260812 07:04:00.551419 10371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:00.580060 10371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:00.581962 10201 heartbeater.cc:503] Master 127.6.61.62:33529 was elected leader, sending a full tablet report...
I20260812 07:04:00.585325 10371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:00.585681 10371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:00.586855 10371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:00.588299 10371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:00.588382 10371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:00.588474 10371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:00.588526 10371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:00.627125 10371 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.3:44211
I20260812 07:04:00.627174 10513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.3:44211 every 8 connection(s)
I20260812 07:04:00.628566 10371 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
I20260812 07:04:00.637159 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 10371
I20260812 07:04:00.637401 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance
I20260812 07:04:00.637962 6388 external_mini_cluster.cc:1428] Setting key 5f8f6817716b18fc76b80803d3416e1f
Aug 12 07:04:00 dist-test-slave-69rg krb5kdc[9918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518240, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:04:00.649873 10516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:00.631529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:51917 (local address 127.6.61.62:33529)
0812 07:04:00.632026 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:00.632031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:00.633059 (+ 1028us) server_negotiation.cc:407] Connection header received
0812 07:04:00.635085 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:00.635091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:00.635183 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:00.635310 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:00.636856 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:00.637809 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:00.639804 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:00.640375 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:00.640518 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:00.644128 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:00.644133 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:00.644136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:00.644169 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:00.646765 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:00.647517 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:00.647524 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:00.647527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:00.647658 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:00.648099 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:00.648107 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:00.648110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:00.648381 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:00.648543 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:00.649217 (+ 674us) server_negotiation.cc:299] Negotiation successful
0812 07:04:00.649448 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":251,"threads_started":1}
I20260812 07:04:00.650781 10519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:00.632109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:33529 (local address 127.6.61.3:51917)
0812 07:04:00.632886 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:00.632933 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:00.634801 (+ 1868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:00.635560 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:00.635567 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:00.636083 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:00.636655 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:00.636672 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:00.638016 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:00.638022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:00.639644 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:00.639663 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:00.640612 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:00.640620 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:00.640908 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:00.641664 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:00.641686 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:00.643913 (+ 2227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:00.646932 (+ 3019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:00.646941 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:00.646956 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:00.647388 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:00.647777 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:00.647784 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:00.647788 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:00.647977 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:00.648517 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:00.648524 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:00.648844 (+ 320us) client_negotiation.cc:769] Sending connection context
0812 07:04:00.649118 (+ 274us) client_negotiation.cc:240] Negotiation successful
0812 07:04:00.649656 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":299,"threads_started":1}
I20260812 07:04:00.652473 10514 heartbeater.cc:347] Connected to a master server at 127.6.61.62:33529
I20260812 07:04:00.652823 10514 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:00.653609 10514 heartbeater.cc:511] Master 127.6.61.62:33529 requested a full tablet report, sending...
I20260812 07:04:00.655224 9976 ts_manager.cc:194] Registered new tserver with Master: 964d7e6fea5e476e9548ee86ec1bb589 (127.6.61.3:44211)
I20260812 07:04:00.656188 9976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.3@KRBTEST.COM'} at 127.6.61.3:51917
I20260812 07:04:00.666107 6388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 07:04:00 dist-test-slave-69rg krb5kdc[9918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518240, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 12 07:04:00 dist-test-slave-69rg krb5kdc[9918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518240, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:04:00.723155 10516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:00.708341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44086 (local address 127.6.61.62:33529)
0812 07:04:00.708638 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:00.708646 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:00.708917 (+ 271us) server_negotiation.cc:407] Connection header received
0812 07:04:00.709076 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:00.709081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:00.709151 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:00.709263 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:00.710797 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:00.711596 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:00.713127 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:00.713610 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:00.713728 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:00.717710 (+ 3982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:00.717715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:00.717719 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:00.717786 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:00.720677 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:00.721355 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:00.721359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:00.721362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:00.721482 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:00.721897 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:00.721901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:00.721903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:00.722222 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:00.722423 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:00.722739 (+ 316us) server_negotiation.cc:299] Negotiation successful
0812 07:04:00.722922 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
W20260812 07:04:00.723852 9976 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44086
I20260812 07:04:00.725904 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 10048
Aug 12 07:04:00 dist-test-slave-69rg krb5kdc[9918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518240, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.61.1@KRBTEST.COM
I20260812 07:04:00.742654 10531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:00.727025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58966 (local address 127.6.61.1:41157)
0812 07:04:00.727702 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:00.727710 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:00.727736 (+ 26us) server_negotiation.cc:407] Connection header received
0812 07:04:00.727794 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:00.727800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:00.728001 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:00.728124 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:00.729312 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:00.730462 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:00.731657 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:00.732274 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:00.732381 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:00.736892 (+ 4511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:00.736904 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:00.736919 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:00.736959 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:00.740104 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:00.740768 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:00.740775 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:00.740777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:00.740896 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:00.741225 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:00.741228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:00.741256 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:00.741586 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:00.741871 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:00.742109 (+ 238us) server_negotiation.cc:299] Negotiation successful
0812 07:04:00.742344 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":363,"threads_started":1}
W20260812 07:04:00.743928 10194 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58966
W20260812 07:04:00.744992 6388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 10048: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 07:04:00.745246 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 10048
I20260812 07:04:00.765975 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 10207
Aug 12 07:04:00 dist-test-slave-69rg krb5kdc[9918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518240, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.61.2@KRBTEST.COM
I20260812 07:04:00.782346 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:00.766962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47418 (local address 127.6.61.2:36969)
0812 07:04:00.767556 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:00.767564 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:00.767592 (+ 28us) server_negotiation.cc:407] Connection header received
0812 07:04:00.767665 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:00.767672 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:00.767893 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:00.768034 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:00.769292 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:00.770490 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:00.771790 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:00.772342 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:00.772446 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:00.776488 (+ 4042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:00.776501 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:00.776516 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:00.776548 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:00.779590 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:00.780247 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:00.780255 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:00.780260 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:00.780392 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:00.780727 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:00.780730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:00.780733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:00.781094 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:00.781367 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:00.781629 (+ 262us) server_negotiation.cc:299] Negotiation successful
0812 07:04:00.782023 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":331,"threads_started":1}
W20260812 07:04:00.783658 10354 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47418
W20260812 07:04:00.784832 6388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 10207: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 07:04:00.784976 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 10207
I20260812 07:04:00.804592 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 10371
Aug 12 07:04:00 dist-test-slave-69rg krb5kdc[9918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518240, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.61.3@KRBTEST.COM
I20260812 07:04:00.821033 10533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:00.805678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44498 (local address 127.6.61.3:44211)
0812 07:04:00.806364 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:00.806371 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:00.806402 (+ 31us) server_negotiation.cc:407] Connection header received
0812 07:04:00.806466 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:00.806473 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:00.806690 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:00.806836 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:00.808142 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:00.809204 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:00.810518 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:00.811072 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:00.811180 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:00.815363 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:00.815374 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:00.815388 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:00.815421 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:00.818452 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:00.819136 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:00.819144 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:00.819146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:00.819273 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:00.819622 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:00.819625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:00.819627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:00.819974 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:00.820210 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:00.820521 (+ 311us) server_negotiation.cc:299] Negotiation successful
0812 07:04:00.820719 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":351,"threads_started":1}
W20260812 07:04:00.822468 10509 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44498
W20260812 07:04:00.823582 6388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 10371: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 07:04:00.823769 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 10371
I20260812 07:04:00.847370 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 9937
I20260812 07:04:00.848654 10010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:00.954932 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 9937
2026-08-12T07:04:00Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3865 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:04:01 dist-test-slave-69rg krb5kdc[10537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:04:01 dist-test-slave-69rg krb5kdc[10537](info): set up 2 sockets
Aug 12 07:04:01 dist-test-slave-69rg krb5kdc[10537](info): commencing operation
krb5kdc: starting...
W20260812 07:04:02.174321 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.176s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:04:02 dist-test-slave-69rg krb5kdc[10537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:04:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:04:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
I20260812 07:04:02.418311 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:32787
--webserver_interface=127.6.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:43509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:32787
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:02.763705 10551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:02.763993 10551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:02.764155 10551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:02.771322 10551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:04:02.771423 10551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:02.771517 10551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:02.771620 10551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:04:02.771704 10551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:04:02.781497 10551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:43509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:32787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:32787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:02.782878 10551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:02.784238 10551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:02.796181 10558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:02.796348 10557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:02.796233 10560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:02.796999 10551 server_base.cc:1034] running on GCE node
I20260812 07:04:02.797868 10551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:02.799377 10551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:02.800592 10551 hybrid_clock.cc:648] HybridClock initialized: now 1786518242800523 us; error 56 us; skew 500 ppm
I20260812 07:04:02.800969 10551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:04:02 dist-test-slave-69rg krb5kdc[10537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518242, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:02.809237 10551 init.cc:377] Logged in from keytab as kudu/127.6.61.62@KRBTEST.COM (short username kudu)
I20260812 07:04:02.810979 10551 webserver.cc:460] Webserver started at http://127.6.61.62:34859/ using document root <none> and password file <none>
I20260812 07:04:02.811683 10551 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:02.811836 10551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:02.812175 10551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:02.815292 10551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "3b29666947aa440d94f5593de6bc351f"
format_stamp: "Formatted at 2026-08-12 07:04:02 on dist-test-slave-69rg"
server_key: "ab9b9da08a99e7856a298deca74e7a7a"
server_key_iv: "7d12fa1b7b6bcc65a6310a13b01d2123"
server_key_version: "encryptionkey@0"
I20260812 07:04:02.816112 10551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "3b29666947aa440d94f5593de6bc351f"
format_stamp: "Formatted at 2026-08-12 07:04:02 on dist-test-slave-69rg"
server_key: "ab9b9da08a99e7856a298deca74e7a7a"
server_key_iv: "7d12fa1b7b6bcc65a6310a13b01d2123"
server_key_version: "encryptionkey@0"
I20260812 07:04:02.821988 10551 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260812 07:04:02.826272 10568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:02.827694 10551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 07:04:02.827898 10551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "3b29666947aa440d94f5593de6bc351f"
format_stamp: "Formatted at 2026-08-12 07:04:02 on dist-test-slave-69rg"
server_key: "ab9b9da08a99e7856a298deca74e7a7a"
server_key_iv: "7d12fa1b7b6bcc65a6310a13b01d2123"
server_key_version: "encryptionkey@0"
I20260812 07:04:02.828115 10551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:02.846658 10551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:02.851052 10551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:02.851402 10551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:02.871069 10551 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:32787
I20260812 07:04:02.871119 10630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:32787 every 8 connection(s)
I20260812 07:04:02.872500 10551 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:04:02.876304 10632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:02.882232 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 10551
I20260812 07:04:02.882514 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:04:02.883230 6388 external_mini_cluster.cc:1428] Setting key 81b1b78aa0b3cdaf4003a7c68d645050
I20260812 07:04:02.888338 10632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b29666947aa440d94f5593de6bc351f: Bootstrap starting.
I20260812 07:04:02.892271 10632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b29666947aa440d94f5593de6bc351f: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:02.893569 10632 log.cc:824] T 00000000000000000000000000000000 P 3b29666947aa440d94f5593de6bc351f: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:02.896950 10632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b29666947aa440d94f5593de6bc351f: No bootstrap required, opened a new log
Aug 12 07:04:02 dist-test-slave-69rg krb5kdc[10537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:04:02.906924 10632 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b29666947aa440d94f5593de6bc351f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b29666947aa440d94f5593de6bc351f" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 32787 } }
I20260812 07:04:02.907426 10632 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b29666947aa440d94f5593de6bc351f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:02.907541 10632 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b29666947aa440d94f5593de6bc351f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b29666947aa440d94f5593de6bc351f, State: Initialized, Role: FOLLOWER
I20260812 07:04:02.908267 10632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b29666947aa440d94f5593de6bc351f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b29666947aa440d94f5593de6bc351f" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 32787 } }
I20260812 07:04:02.908605 10632 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b29666947aa440d94f5593de6bc351f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:04:02.908782 10632 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b29666947aa440d94f5593de6bc351f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:04:02.908967 10632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b29666947aa440d94f5593de6bc351f [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:02.910647 10635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:02.885636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53674 (local address 127.6.61.62:32787)
0812 07:04:02.886603 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:02.886619 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:02.886673 (+ 54us) server_negotiation.cc:407] Connection header received
0812 07:04:02.889137 (+ 2464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:02.889169 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:02.889668 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:02.890145 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:02.891561 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:02.893629 (+ 2068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:02.895519 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:02.896571 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:02.896748 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:02.901315 (+ 4567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:02.901331 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:02.901350 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:02.901414 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:02.905632 (+ 4218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:02.906466 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:02.906477 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:02.906487 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:02.906700 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:02.907103 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:02.907108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:02.907112 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:02.907894 (+ 782us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:02.908197 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:02.908709 (+ 512us) server_negotiation.cc:299] Negotiation successful
0812 07:04:02.909187 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":410,"threads_started":1}
I20260812 07:04:02.911993 10632 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b29666947aa440d94f5593de6bc351f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b29666947aa440d94f5593de6bc351f" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 32787 } }
I20260812 07:04:02.912593 10632 leader_election.cc:302] T 00000000000000000000000000000000 P 3b29666947aa440d94f5593de6bc351f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b29666947aa440d94f5593de6bc351f; no voters:
I20260812 07:04:02.913236 10632 leader_election.cc:288] T 00000000000000000000000000000000 P 3b29666947aa440d94f5593de6bc351f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:04:02.913383 10637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b29666947aa440d94f5593de6bc351f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:04:02.914611 10637 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b29666947aa440d94f5593de6bc351f [term 1 LEADER]: Becoming Leader. State: Replica: 3b29666947aa440d94f5593de6bc351f, State: Running, Role: LEADER
I20260812 07:04:02.915110 10637 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b29666947aa440d94f5593de6bc351f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b29666947aa440d94f5593de6bc351f" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 32787 } }
I20260812 07:04:02.915393 10632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b29666947aa440d94f5593de6bc351f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:04:02.917907 10639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b29666947aa440d94f5593de6bc351f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b29666947aa440d94f5593de6bc351f. Latest consensus state: current_term: 1 leader_uuid: "3b29666947aa440d94f5593de6bc351f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b29666947aa440d94f5593de6bc351f" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 32787 } } }
I20260812 07:04:02.918234 10638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b29666947aa440d94f5593de6bc351f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b29666947aa440d94f5593de6bc351f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b29666947aa440d94f5593de6bc351f" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 32787 } } }
I20260812 07:04:02.918473 10639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b29666947aa440d94f5593de6bc351f [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:02.918531 10638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b29666947aa440d94f5593de6bc351f [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:02.920351 10645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:04:02.926451 10645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:04:02.935762 10645 catalog_manager.cc:1347] Generated new cluster ID: b1909373d1e04beb8031fe36434430b7
I20260812 07:04:02.935892 10645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:04:02.956604 10645 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:04:02.958024 10645 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:04:02.967420 10645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b29666947aa440d94f5593de6bc351f: Generated new TSK 0
I20260812 07:04:02.968281 10645 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
I20260812 07:04:03.034672 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.1:0
--local_ip_for_outbound_sockets=127.6.61.1
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:32787
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:43509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:03.378578 10658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:03.378872 10658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:03.379037 10658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:03.386278 10658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:03.386387 10658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:03.386564 10658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.1
I20260812 07:04:03.396512 10658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:43509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:32787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.1
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:03.397987 10658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:03.399376 10658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:03.412672 10665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:03.412716 10666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:03.412744 10668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:03.413686 10658 server_base.cc:1034] running on GCE node
I20260812 07:04:03.414348 10658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:03.415316 10658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:03.416540 10658 hybrid_clock.cc:648] HybridClock initialized: now 1786518243416497 us; error 56 us; skew 500 ppm
I20260812 07:04:03.416823 10658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:04:03 dist-test-slave-69rg krb5kdc[10537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518243, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:03.420164 10658 init.cc:377] Logged in from keytab as kudu/127.6.61.1@KRBTEST.COM (short username kudu)
I20260812 07:04:03.421593 10658 webserver.cc:460] Webserver started at http://127.6.61.1:46629/ using document root <none> and password file <none>
I20260812 07:04:03.422349 10658 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:03.422490 10658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:03.422819 10658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:03.426075 10658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-0/data/instance:
uuid: "8b767e61d81447278597d40202516dfa"
format_stamp: "Formatted at 2026-08-12 07:04:03 on dist-test-slave-69rg"
server_key: "823b1b07e354ec45c853dde2390ad838"
server_key_iv: "1812ee85a3dd42671a2d80014622fcd6"
server_key_version: "encryptionkey@0"
I20260812 07:04:03.426900 10658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance:
uuid: "8b767e61d81447278597d40202516dfa"
format_stamp: "Formatted at 2026-08-12 07:04:03 on dist-test-slave-69rg"
server_key: "823b1b07e354ec45c853dde2390ad838"
server_key_iv: "1812ee85a3dd42671a2d80014622fcd6"
server_key_version: "encryptionkey@0"
I20260812 07:04:03.432124 10658 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 07:04:03.435755 10678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:03.437079 10658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 07:04:03.437256 10658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-0/wal
uuid: "8b767e61d81447278597d40202516dfa"
format_stamp: "Formatted at 2026-08-12 07:04:03 on dist-test-slave-69rg"
server_key: "823b1b07e354ec45c853dde2390ad838"
server_key_iv: "1812ee85a3dd42671a2d80014622fcd6"
server_key_version: "encryptionkey@0"
I20260812 07:04:03.437474 10658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:03.455193 10658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:03.459676 10658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:03.459990 10658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:03.460964 10658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:03.462375 10658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:03.462451 10658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:03.462607 10658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:03.462678 10658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:03.497241 10658 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.1:34061
I20260812 07:04:03.497287 10807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.1:34061 every 8 connection(s)
I20260812 07:04:03.498647 10658 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
I20260812 07:04:03.499763 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 10658
I20260812 07:04:03.500015 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance
I20260812 07:04:03.500722 6388 external_mini_cluster.cc:1428] Setting key a811312dc97ec66fe279f7c81320f212
Aug 12 07:04:03 dist-test-slave-69rg krb5kdc[10537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518243, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:04:03.520941 10811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:03.501444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:54793 (local address 127.6.61.62:32787)
0812 07:04:03.501975 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:03.501982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:03.503055 (+ 1073us) server_negotiation.cc:407] Connection header received
0812 07:04:03.504925 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:03.504933 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:03.505028 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:03.505163 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:03.506706 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:03.507874 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:03.509450 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:03.510205 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:03.510343 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:03.514927 (+ 4584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:03.514934 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:03.514939 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:03.514981 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:03.517720 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:03.518523 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:03.518530 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:03.518534 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:03.518666 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:03.519157 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:03.519161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:03.519163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:03.519428 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:03.519600 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:03.520345 (+ 745us) server_negotiation.cc:299] Negotiation successful
0812 07:04:03.520577 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":245,"threads_started":1}
I20260812 07:04:03.522023 10812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:03.502052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:32787 (local address 127.6.61.1:54793)
0812 07:04:03.502899 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:03.502945 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:03.504638 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:03.505363 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:03.505371 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:03.505965 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:03.506521 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:03.506538 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:03.508050 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:03.508053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:03.509293 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:03.509306 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:03.510587 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:03.510599 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:03.510954 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:03.511798 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:03.511839 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:03.514705 (+ 2866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:03.517884 (+ 3179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:03.517895 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:03.517911 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:03.518381 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:03.518799 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:03.518805 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:03.518809 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:03.519016 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:03.519570 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:03.519578 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:03.519942 (+ 364us) client_negotiation.cc:769] Sending connection context
0812 07:04:03.520226 (+ 284us) client_negotiation.cc:240] Negotiation successful
0812 07:04:03.520773 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":372,"threads_started":1}
I20260812 07:04:03.523877 10808 heartbeater.cc:347] Connected to a master server at 127.6.61.62:32787
I20260812 07:04:03.524255 10808 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:03.524997 10808 heartbeater.cc:511] Master 127.6.61.62:32787 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.61.2@KRBTEST.COM; defaulting to no policy
I20260812 07:04:03.527410 10591 ts_manager.cc:194] Registered new tserver with Master: 8b767e61d81447278597d40202516dfa (127.6.61.1:34061)
I20260812 07:04:03.528885 10591 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.1@KRBTEST.COM'} at 127.6.61.1:54793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
I20260812 07:04:03.558660 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.2:0
--local_ip_for_outbound_sockets=127.6.61.2
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:32787
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:43509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:03.908340 10815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:03.908636 10815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:03.908802 10815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:03.916183 10815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:03.916296 10815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:03.916486 10815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.2
I20260812 07:04:03.926432 10815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:43509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:32787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.2
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:03.927939 10815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:03.929301 10815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:03.943310 10823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:03.943567 10825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:03.943583 10822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:03.944360 10815 server_base.cc:1034] running on GCE node
I20260812 07:04:03.944943 10815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:03.945935 10815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:03.947206 10815 hybrid_clock.cc:648] HybridClock initialized: now 1786518243947135 us; error 57 us; skew 500 ppm
I20260812 07:04:03.947587 10815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:04:03 dist-test-slave-69rg krb5kdc[10537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518243, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:03.950973 10815 init.cc:377] Logged in from keytab as kudu/127.6.61.2@KRBTEST.COM (short username kudu)
I20260812 07:04:03.952564 10815 webserver.cc:460] Webserver started at http://127.6.61.2:36749/ using document root <none> and password file <none>
I20260812 07:04:03.953295 10815 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:03.953440 10815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:03.953776 10815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:03.956812 10815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-1/data/instance:
uuid: "ab254dcf669d4788b7f84473f2b081b5"
format_stamp: "Formatted at 2026-08-12 07:04:03 on dist-test-slave-69rg"
server_key: "44122d0e2026ef098e0c9d885190da0c"
server_key_iv: "eba7930a610cfdaae68861f56d913116"
server_key_version: "encryptionkey@0"
I20260812 07:04:03.957662 10815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance:
uuid: "ab254dcf669d4788b7f84473f2b081b5"
format_stamp: "Formatted at 2026-08-12 07:04:03 on dist-test-slave-69rg"
server_key: "44122d0e2026ef098e0c9d885190da0c"
server_key_iv: "eba7930a610cfdaae68861f56d913116"
server_key_version: "encryptionkey@0"
I20260812 07:04:03.963117 10815 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 07:04:03.966789 10834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:03.968348 10815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 07:04:03.968494 10815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-1/wal
uuid: "ab254dcf669d4788b7f84473f2b081b5"
format_stamp: "Formatted at 2026-08-12 07:04:03 on dist-test-slave-69rg"
server_key: "44122d0e2026ef098e0c9d885190da0c"
server_key_iv: "eba7930a610cfdaae68861f56d913116"
server_key_version: "encryptionkey@0"
I20260812 07:04:03.968678 10815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:03.982671 10815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:03.987136 10815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:03.987396 10815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:03.988348 10815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:03.989626 10815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:03.989699 10815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:03.989782 10815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:03.989883 10815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260812 07:04:04.025770 10815 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.2:43807
I20260812 07:04:04.025906 10968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.2:43807 every 8 connection(s)
I20260812 07:04:04.027158 10815 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
I20260812 07:04:04.032546 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 10815
I20260812 07:04:04.032812 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance
I20260812 07:04:04.033376 6388 external_mini_cluster.cc:1428] Setting key 6e3807240a0cc523a426b7a27bbaf026
Aug 12 07:04:04 dist-test-slave-69rg krb5kdc[10537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518243, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:04:04.052192 10971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:04.030183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:46675 (local address 127.6.61.62:32787)
0812 07:04:04.030675 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:04.030680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:04.031501 (+ 821us) server_negotiation.cc:407] Connection header received
0812 07:04:04.034307 (+ 2806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:04.034313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:04.034416 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:04.034548 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:04.036340 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:04.037147 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:04.039779 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:04.040300 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:04.040421 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:04.045237 (+ 4816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:04.045244 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:04.045248 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:04.045287 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:04.048495 (+ 3208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:04.049398 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:04.049403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:04.049406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:04.049547 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:04.050163 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:04.050168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:04.050170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:04.050471 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:04.050655 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:04.051478 (+ 823us) server_negotiation.cc:299] Negotiation successful
0812 07:04:04.051740 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":258,"threads_started":1}
I20260812 07:04:04.053365 10972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:04.030519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:32787 (local address 127.6.61.2:46675)
0812 07:04:04.031325 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:04.031374 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:04.033966 (+ 2592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:04.034813 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:04.034822 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:04.035371 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:04.036142 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:04.036168 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:04.037336 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:04.037345 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:04.039596 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:04.039616 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:04.040586 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:04.040597 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:04.040951 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:04.041812 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:04.041881 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:04.044943 (+ 3062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:04.048700 (+ 3757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:04.048711 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:04.048734 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:04.049223 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:04.049687 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:04.049693 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:04.049697 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:04.050008 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:04.050624 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:04.050642 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:04.051047 (+ 405us) client_negotiation.cc:769] Sending connection context
0812 07:04:04.051355 (+ 308us) client_negotiation.cc:240] Negotiation successful
0812 07:04:04.051992 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":294,"threads_started":1}
I20260812 07:04:04.055089 10969 heartbeater.cc:347] Connected to a master server at 127.6.61.62:32787
I20260812 07:04:04.055460 10969 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:04.056306 10969 heartbeater.cc:511] Master 127.6.61.62:32787 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.61.3@KRBTEST.COM; defaulting to no policy
I20260812 07:04:04.058435 10591 ts_manager.cc:194] Registered new tserver with Master: ab254dcf669d4788b7f84473f2b081b5 (127.6.61.2:43807)
I20260812 07:04:04.059508 10591 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.2@KRBTEST.COM'} at 127.6.61.2:46675
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
I20260812 07:04:04.092978 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.3:0
--local_ip_for_outbound_sockets=127.6.61.3
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:32787
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:43509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:04.432786 10976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:04.433058 10976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:04.433224 10976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:04.440389 10976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:04.440505 10976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:04.440698 10976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.3
I20260812 07:04:04.450877 10976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:43509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:32787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.3
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:04.452219 10976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:04.453560 10976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:04.467443 10981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:04.467855 10984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:04.467911 10982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:04.468854 10976 server_base.cc:1034] running on GCE node
I20260812 07:04:04.469516 10976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:04.470566 10976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:04.471836 10976 hybrid_clock.cc:648] HybridClock initialized: now 1786518244471781 us; error 59 us; skew 500 ppm
I20260812 07:04:04.472270 10976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:04:04 dist-test-slave-69rg krb5kdc[10537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518244, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:04.475787 10976 init.cc:377] Logged in from keytab as kudu/127.6.61.3@KRBTEST.COM (short username kudu)
I20260812 07:04:04.477268 10976 webserver.cc:460] Webserver started at http://127.6.61.3:44527/ using document root <none> and password file <none>
I20260812 07:04:04.477999 10976 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:04.478129 10976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:04.478399 10976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:04.481304 10976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-2/data/instance:
uuid: "d8a24d1a0e474ae9a1747fa12339c804"
format_stamp: "Formatted at 2026-08-12 07:04:04 on dist-test-slave-69rg"
server_key: "4ce0e4835b92fa2b171d5d16c9616d2a"
server_key_iv: "2ebdf036a95730f5cb1ee519165a0a8f"
server_key_version: "encryptionkey@0"
I20260812 07:04:04.482128 10976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance:
uuid: "d8a24d1a0e474ae9a1747fa12339c804"
format_stamp: "Formatted at 2026-08-12 07:04:04 on dist-test-slave-69rg"
server_key: "4ce0e4835b92fa2b171d5d16c9616d2a"
server_key_iv: "2ebdf036a95730f5cb1ee519165a0a8f"
server_key_version: "encryptionkey@0"
I20260812 07:04:04.487308 10976 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260812 07:04:04.491000 10992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:04.492462 10976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 07:04:04.492704 10976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-2/wal
uuid: "d8a24d1a0e474ae9a1747fa12339c804"
format_stamp: "Formatted at 2026-08-12 07:04:04 on dist-test-slave-69rg"
server_key: "4ce0e4835b92fa2b171d5d16c9616d2a"
server_key_iv: "2ebdf036a95730f5cb1ee519165a0a8f"
server_key_version: "encryptionkey@0"
I20260812 07:04:04.492954 10976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:04.526005 10976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:04.530755 10976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:04.531032 10976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:04.532119 10976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:04.532220 10808 heartbeater.cc:503] Master 127.6.61.62:32787 was elected leader, sending a full tablet report...
I20260812 07:04:04.534058 10976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:04.534170 10976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:04.534349 10976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:04.534446 10976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:04.572800 10976 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.3:32961
I20260812 07:04:04.572858 11122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.3:32961 every 8 connection(s)
I20260812 07:04:04.574322 10976 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
I20260812 07:04:04.577625 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 10976
I20260812 07:04:04.577895 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance
I20260812 07:04:04.578504 6388 external_mini_cluster.cc:1428] Setting key 66cacea971b8d0013d37773ce34b4700
Aug 12 07:04:04 dist-test-slave-69rg krb5kdc[10537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518244, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:04:04.596127 11125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:04.577083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:43837 (local address 127.6.61.62:32787)
0812 07:04:04.577747 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:04.577755 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:04.578977 (+ 1222us) server_negotiation.cc:407] Connection header received
0812 07:04:04.581459 (+ 2482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:04.581465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:04.581585 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:04.581735 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:04.583447 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:04.584334 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:04.585945 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:04.586381 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:04.586488 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:04.589967 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:04.589972 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:04.589976 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:04.590010 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:04.592528 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:04.593473 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:04.593478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:04.593481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:04.593634 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:04.594252 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:04.594257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:04.594260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:04.594596 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:04.594774 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:04.595497 (+ 723us) server_negotiation.cc:299] Negotiation successful
0812 07:04:04.595737 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":330,"threads_started":1}
I20260812 07:04:04.597222 11127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:04.577715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:32787 (local address 127.6.61.3:43837)
0812 07:04:04.578775 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:04.578826 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:04.581155 (+ 2329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:04.582044 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:04.582051 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:04.582697 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:04.583276 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:04.583293 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:04.584474 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:04.584477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:04.585719 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:04.585732 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:04.586585 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:04.586590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:04.586810 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:04.587441 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:04.587463 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:04.589709 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:04.592682 (+ 2973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:04.592693 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:04.592716 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:04.593285 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:04.593775 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:04.593780 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:04.593783 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:04.594070 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:04.594765 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:04.594773 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:04.595108 (+ 335us) client_negotiation.cc:769] Sending connection context
0812 07:04:04.595375 (+ 267us) client_negotiation.cc:240] Negotiation successful
0812 07:04:04.595928 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"thread_start_us":478,"threads_started":1}
I20260812 07:04:04.598708 11123 heartbeater.cc:347] Connected to a master server at 127.6.61.62:32787
I20260812 07:04:04.599030 11123 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:04.599761 11123 heartbeater.cc:511] Master 127.6.61.62:32787 requested a full tablet report, sending...
I20260812 07:04:04.601230 10591 ts_manager.cc:194] Registered new tserver with Master: d8a24d1a0e474ae9a1747fa12339c804 (127.6.61.3:32961)
I20260812 07:04:04.602162 10591 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.3@KRBTEST.COM'} at 127.6.61.3:43837
I20260812 07:04:04.607326 6388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 07:04:04 dist-test-slave-69rg krb5kdc[10537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 07:04:04 dist-test-slave-69rg krb5kdc[10537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.1@KRBTEST.COM
I20260812 07:04:04.664615 11132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:04.648252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48092 (local address 127.6.61.1:34061)
0812 07:04:04.648992 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:04.649001 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:04.649035 (+ 34us) server_negotiation.cc:407] Connection header received
0812 07:04:04.649106 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:04.649113 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:04.649344 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:04.649501 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:04.650954 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:04.652042 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:04.653294 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:04.653927 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:04.654034 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:04.658364 (+ 4330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:04.658378 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:04.658395 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:04.658433 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:04.661653 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:04.662420 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:04.662427 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:04.662429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:04.662562 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:04.662988 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:04.662991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:04.662994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:04.663387 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:04.663745 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:04.663987 (+ 242us) server_negotiation.cc:299] Negotiation successful
0812 07:04:04.664202 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":384,"threads_started":1}
Aug 12 07:04:04 dist-test-slave-69rg krb5kdc[10537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:04:04.684193 11125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:04.669555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53678 (local address 127.6.61.62:32787)
0812 07:04:04.669794 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:04.669799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:04.669932 (+ 133us) server_negotiation.cc:407] Connection header received
0812 07:04:04.670071 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:04.670075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:04.670144 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:04.670227 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:04.671514 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:04.672295 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:04.673628 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:04.674138 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:04.674252 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:04.678764 (+ 4512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:04.678770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:04.678773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:04.678811 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:04.681392 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:04.682293 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:04.682297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:04.682299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:04.682420 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:04.682851 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:04.682854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:04.682856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:04.683182 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:04.683386 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:04.683715 (+ 329us) server_negotiation.cc:299] Negotiation successful
0812 07:04:04.683896 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260812 07:04:04.684911 10591 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53678
I20260812 07:04:04.686168 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 10658
I20260812 07:04:04.700941 11132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:04.687272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48100 (local address 127.6.61.1:34061)
0812 07:04:04.687524 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:04.687531 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:04.687716 (+ 185us) server_negotiation.cc:407] Connection header received
0812 07:04:04.687846 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:04.687850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:04.687920 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:04.688006 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:04.689233 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:04.690082 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:04.691348 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:04.691841 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:04.691961 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:04.694795 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:04.694801 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:04.694804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:04.694844 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:04.698267 (+ 3423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:04.698927 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:04.698932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:04.698935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:04.699084 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:04.699458 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:04.699463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:04.699465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:04.699766 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:04.699965 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:04.700236 (+ 271us) server_negotiation.cc:299] Negotiation successful
0812 07:04:04.700421 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 07:04:04.701720 10803 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:04.786051 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 10658
I20260812 07:04:04.806012 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 10815
Aug 12 07:04:04 dist-test-slave-69rg krb5kdc[10537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.2@KRBTEST.COM
I20260812 07:04:04.823689 11139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:04.807056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48104 (local address 127.6.61.2:43807)
0812 07:04:04.807620 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:04.807628 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:04.807655 (+ 27us) server_negotiation.cc:407] Connection header received
0812 07:04:04.807725 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:04.807732 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:04.807951 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:04.808083 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:04.809294 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:04.810448 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:04.811674 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:04.812229 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:04.812338 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:04.817258 (+ 4920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:04.817274 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:04.817291 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:04.817332 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:04.820527 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:04.821317 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:04.821324 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:04.821326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:04.821462 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:04.821917 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:04.821921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:04.821923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:04.822326 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:04.822664 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:04.823018 (+ 354us) server_negotiation.cc:299] Negotiation successful
0812 07:04:04.823246 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":293,"threads_started":1}
I20260812 07:04:04.825047 10964 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:04.910240 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 10815
I20260812 07:04:04.932710 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 10976
Aug 12 07:04:04 dist-test-slave-69rg krb5kdc[10537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.3@KRBTEST.COM
I20260812 07:04:04.952090 11141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:04.934033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46428 (local address 127.6.61.3:32961)
0812 07:04:04.934864 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:04.934872 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:04.934905 (+ 33us) server_negotiation.cc:407] Connection header received
0812 07:04:04.934973 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:04.934980 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:04.935258 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:04.935427 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:04.936783 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:04.938052 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:04.940013 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:04.940670 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:04.940792 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:04.945617 (+ 4825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:04.945631 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:04.945648 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:04.945688 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:04.949139 (+ 3451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:04.949930 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:04.949938 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:04.949941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:04.950081 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:04.950504 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:04.950508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:04.950511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:04.950891 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:04.951172 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:04.951439 (+ 267us) server_negotiation.cc:299] Negotiation successful
0812 07:04:04.951695 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":486,"threads_started":1}
I20260812 07:04:04.953516 11118 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:05.039004 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 10976
I20260812 07:04:05.060317 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 10551
I20260812 07:04:05.061699 10626 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:05.167137 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 10551
2026-08-12T07:04:05Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4214 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:04:05 dist-test-slave-69rg krb5kdc[11146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:04:05 dist-test-slave-69rg krb5kdc[11146](info): set up 2 sockets
Aug 12 07:04:05 dist-test-slave-69rg krb5kdc[11146](info): commencing operation
krb5kdc: starting...
W20260812 07:04:06.412180 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.198s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:04:06 dist-test-slave-69rg krb5kdc[11146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:04:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:04:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
I20260812 07:04:06.659555 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:35795
--webserver_interface=127.6.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:34203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:35795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:06.998488 11162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:06.998852 11162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:06.999043 11162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:07.008814 11162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:04:07.008988 11162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:07.009074 11162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:07.009186 11162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:04:07.009263 11162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:04:07.020182 11162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:34203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:35795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:35795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:07.021587 11162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:07.023011 11162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:07.035195 11169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:07.035197 11168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:07.035321 11172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:07.036420 11162 server_base.cc:1034] running on GCE node
I20260812 07:04:07.037359 11162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:07.038976 11162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:07.040251 11162 hybrid_clock.cc:648] HybridClock initialized: now 1786518247040193 us; error 99 us; skew 500 ppm
I20260812 07:04:07.040673 11162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:04:07 dist-test-slave-69rg krb5kdc[11146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518247, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:07.044153 11162 init.cc:377] Logged in from keytab as kudu/127.6.61.62@KRBTEST.COM (short username kudu)
I20260812 07:04:07.045508 11162 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:07.045657 11162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:07.046070 11162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:07.049235 11162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "206b848b010543aa9e5e4b597a19285c"
format_stamp: "Formatted at 2026-08-12 07:04:07 on dist-test-slave-69rg"
server_key: "5d25c28953c289f7ebc6e38fe1c12034"
server_key_iv: "a1f6a80d3ef6d963347889789a40da79"
server_key_version: "encryptionkey@0"
I20260812 07:04:07.050225 11162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "206b848b010543aa9e5e4b597a19285c"
format_stamp: "Formatted at 2026-08-12 07:04:07 on dist-test-slave-69rg"
server_key: "5d25c28953c289f7ebc6e38fe1c12034"
server_key_iv: "a1f6a80d3ef6d963347889789a40da79"
server_key_version: "encryptionkey@0"
I20260812 07:04:07.055855 11162 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 07:04:07.059530 11180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:07.060838 11162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260812 07:04:07.061041 11162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "206b848b010543aa9e5e4b597a19285c"
format_stamp: "Formatted at 2026-08-12 07:04:07 on dist-test-slave-69rg"
server_key: "5d25c28953c289f7ebc6e38fe1c12034"
server_key_iv: "a1f6a80d3ef6d963347889789a40da79"
server_key_version: "encryptionkey@0"
I20260812 07:04:07.061288 11162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:07.075807 11162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:07.080696 11162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:07.081025 11162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:07.101997 11162 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:35795
I20260812 07:04:07.102038 11243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:35795 every 8 connection(s)
I20260812 07:04:07.103004 11162 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:04:07.103644 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 11162
I20260812 07:04:07.103935 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:04:07.104701 6388 external_mini_cluster.cc:1428] Setting key 770fe8a379e8a3ddc1ecc9a5cbeb0a1e
I20260812 07:04:07.107540 11244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 07:04:07 dist-test-slave-69rg krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:04:07.120591 11244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 206b848b010543aa9e5e4b597a19285c: Bootstrap starting.
I20260812 07:04:07.124967 11244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 206b848b010543aa9e5e4b597a19285c: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:07.126322 11244 log.cc:824] T 00000000000000000000000000000000 P 206b848b010543aa9e5e4b597a19285c: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:07.129869 11244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 206b848b010543aa9e5e4b597a19285c: No bootstrap required, opened a new log
I20260812 07:04:07.130807 11247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:07.106816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59656 (local address 127.6.61.62:35795)
0812 07:04:07.107683 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:07.107699 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:07.107761 (+ 62us) server_negotiation.cc:407] Connection header received
0812 07:04:07.110138 (+ 2377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:07.110167 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:07.110673 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:07.111116 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:07.112544 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:07.114652 (+ 2108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:07.116006 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:07.117073 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:07.117259 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:07.121564 (+ 4305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:07.121575 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:07.121592 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:07.121638 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:07.125797 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:07.126661 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:07.126671 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:07.126679 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:07.126955 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:07.127438 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:07.127442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:07.127444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:07.128198 (+ 754us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:07.128477 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:07.128915 (+ 438us) server_negotiation.cc:299] Negotiation successful
0812 07:04:07.129366 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":365,"threads_started":1}
I20260812 07:04:07.138146 11244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 206b848b010543aa9e5e4b597a19285c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "206b848b010543aa9e5e4b597a19285c" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35795 } }
I20260812 07:04:07.138635 11244 raft_consensus.cc:374] T 00000000000000000000000000000000 P 206b848b010543aa9e5e4b597a19285c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:07.138720 11244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 206b848b010543aa9e5e4b597a19285c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 206b848b010543aa9e5e4b597a19285c, State: Initialized, Role: FOLLOWER
I20260812 07:04:07.139184 11244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 206b848b010543aa9e5e4b597a19285c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "206b848b010543aa9e5e4b597a19285c" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35795 } }
I20260812 07:04:07.139398 11244 raft_consensus.cc:388] T 00000000000000000000000000000000 P 206b848b010543aa9e5e4b597a19285c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:04:07.139506 11244 raft_consensus.cc:482] T 00000000000000000000000000000000 P 206b848b010543aa9e5e4b597a19285c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:04:07.139633 11244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 206b848b010543aa9e5e4b597a19285c [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:07.141739 11244 raft_consensus.cc:504] T 00000000000000000000000000000000 P 206b848b010543aa9e5e4b597a19285c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "206b848b010543aa9e5e4b597a19285c" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35795 } }
I20260812 07:04:07.142329 11244 leader_election.cc:302] T 00000000000000000000000000000000 P 206b848b010543aa9e5e4b597a19285c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 206b848b010543aa9e5e4b597a19285c; no voters:
I20260812 07:04:07.142968 11244 leader_election.cc:288] T 00000000000000000000000000000000 P 206b848b010543aa9e5e4b597a19285c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:04:07.143179 11251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 206b848b010543aa9e5e4b597a19285c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:04:07.144014 11251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 206b848b010543aa9e5e4b597a19285c [term 1 LEADER]: Becoming Leader. State: Replica: 206b848b010543aa9e5e4b597a19285c, State: Running, Role: LEADER
I20260812 07:04:07.144636 11251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 206b848b010543aa9e5e4b597a19285c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "206b848b010543aa9e5e4b597a19285c" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35795 } }
I20260812 07:04:07.145364 11244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 206b848b010543aa9e5e4b597a19285c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:04:07.147838 11253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 206b848b010543aa9e5e4b597a19285c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 206b848b010543aa9e5e4b597a19285c. Latest consensus state: current_term: 1 leader_uuid: "206b848b010543aa9e5e4b597a19285c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "206b848b010543aa9e5e4b597a19285c" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35795 } } }
I20260812 07:04:07.148319 11253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 206b848b010543aa9e5e4b597a19285c [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:07.147838 11252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 206b848b010543aa9e5e4b597a19285c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "206b848b010543aa9e5e4b597a19285c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "206b848b010543aa9e5e4b597a19285c" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35795 } } }
I20260812 07:04:07.148986 11252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 206b848b010543aa9e5e4b597a19285c [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:07.152184 11261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:04:07.157109 11261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:04:07.167438 11261 catalog_manager.cc:1347] Generated new cluster ID: 60dba08f8f52413091466f8312a16590
I20260812 07:04:07.167568 11261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:04:07.181979 11261 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:04:07.183387 11261 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:04:07.189949 11261 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 206b848b010543aa9e5e4b597a19285c: Generated new TSK 0
I20260812 07:04:07.191009 11261 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
I20260812 07:04:07.257710 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.1:0
--local_ip_for_outbound_sockets=127.6.61.1
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:35795
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:34203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:07.607326 11277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:07.607641 11277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:07.607743 11277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:07.615307 11277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:07.615422 11277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:07.615617 11277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.1
I20260812 07:04:07.625660 11277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:34203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:35795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.1
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:07.627038 11277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:07.628407 11277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:07.642025 11283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:07.642102 11288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:07.642026 11285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:07.643013 11277 server_base.cc:1034] running on GCE node
I20260812 07:04:07.643803 11277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:07.644804 11277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:07.646005 11277 hybrid_clock.cc:648] HybridClock initialized: now 1786518247645956 us; error 39 us; skew 500 ppm
I20260812 07:04:07.646376 11277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:04:07 dist-test-slave-69rg krb5kdc[11146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518247, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:07.649777 11277 init.cc:377] Logged in from keytab as kudu/127.6.61.1@KRBTEST.COM (short username kudu)
I20260812 07:04:07.651154 11277 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:07.651296 11277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:07.651731 11277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:07.654896 11277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-0/data/instance:
uuid: "e8dc27683e6c4188bbe0ec4c397b8796"
format_stamp: "Formatted at 2026-08-12 07:04:07 on dist-test-slave-69rg"
server_key: "c2b78c37c048fc429d300a875cf1534b"
server_key_iv: "79026824b5dbcb2e6a46bf48e3f29bc5"
server_key_version: "encryptionkey@0"
I20260812 07:04:07.655819 11277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance:
uuid: "e8dc27683e6c4188bbe0ec4c397b8796"
format_stamp: "Formatted at 2026-08-12 07:04:07 on dist-test-slave-69rg"
server_key: "c2b78c37c048fc429d300a875cf1534b"
server_key_iv: "79026824b5dbcb2e6a46bf48e3f29bc5"
server_key_version: "encryptionkey@0"
I20260812 07:04:07.661022 11277 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260812 07:04:07.664620 11295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:07.666131 11277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:04:07.666348 11277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-0/wal
uuid: "e8dc27683e6c4188bbe0ec4c397b8796"
format_stamp: "Formatted at 2026-08-12 07:04:07 on dist-test-slave-69rg"
server_key: "c2b78c37c048fc429d300a875cf1534b"
server_key_iv: "79026824b5dbcb2e6a46bf48e3f29bc5"
server_key_version: "encryptionkey@0"
I20260812 07:04:07.666589 11277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:07.685300 11277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:07.690512 11277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:07.690863 11277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:07.691905 11277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:07.693514 11277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:07.693648 11277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:07.693797 11277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:07.693925 11277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260812 07:04:07.730803 11277 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.1:37597
I20260812 07:04:07.730859 11435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.1:37597 every 8 connection(s)
I20260812 07:04:07.731745 11277 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
I20260812 07:04:07.734025 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 11277
I20260812 07:04:07.734308 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance
I20260812 07:04:07.735312 6388 external_mini_cluster.cc:1428] Setting key e89da61dea62d668b71a20ad76db7961
Aug 12 07:04:07 dist-test-slave-69rg krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518247, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:04:07.755787 11438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:07.734696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:60787 (local address 127.6.61.62:35795)
0812 07:04:07.735251 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:07.735256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:07.736596 (+ 1340us) server_negotiation.cc:407] Connection header received
0812 07:04:07.738981 (+ 2385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:07.738987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:07.739084 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:07.739212 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:07.740819 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:07.741559 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:07.743349 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:07.743746 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:07.743896 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:07.748000 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:07.748010 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:07.748018 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:07.748065 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:07.751792 (+ 3727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:07.752751 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:07.752760 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:07.752766 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:07.752922 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:07.753501 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:07.753508 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:07.753514 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:07.753975 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:07.754146 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:07.754953 (+ 807us) server_negotiation.cc:299] Negotiation successful
0812 07:04:07.755241 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":320,"threads_started":1}
I20260812 07:04:07.756546 11441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:07.735396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:35795 (local address 127.6.61.1:60787)
0812 07:04:07.736396 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:07.736463 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:07.738682 (+ 2219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:07.739471 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:07.739479 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:07.740026 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:07.740631 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:07.740648 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:07.741728 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:07.741735 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:07.743170 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:07.743184 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:07.744008 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:07.744017 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:07.744257 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:07.744990 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:07.745012 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:07.747742 (+ 2730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:07.751971 (+ 4229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:07.751979 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:07.751994 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:07.752595 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:07.753113 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:07.753119 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:07.753124 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:07.753353 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:07.754136 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:07.754143 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:07.754507 (+ 364us) client_negotiation.cc:769] Sending connection context
0812 07:04:07.754790 (+ 283us) client_negotiation.cc:240] Negotiation successful
0812 07:04:07.755241 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":417,"threads_started":1}
I20260812 07:04:07.758543 11436 heartbeater.cc:347] Connected to a master server at 127.6.61.62:35795
I20260812 07:04:07.758939 11436 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:07.759845 11436 heartbeater.cc:511] Master 127.6.61.62:35795 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.61.2@KRBTEST.COM; defaulting to no policy
I20260812 07:04:07.762596 11200 ts_manager.cc:194] Registered new tserver with Master: e8dc27683e6c4188bbe0ec4c397b8796 (127.6.61.1:37597)
I20260812 07:04:07.764387 11200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.1@KRBTEST.COM'} at 127.6.61.1:60787
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
I20260812 07:04:07.797256 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.2:0
--local_ip_for_outbound_sockets=127.6.61.2
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:35795
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:34203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:08.145027 11445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:08.145356 11445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:08.145466 11445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:08.153487 11445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:08.153622 11445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:08.153743 11445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.2
I20260812 07:04:08.165151 11445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:34203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:35795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.2
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:08.166854 11445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:08.168152 11445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:08.182157 11452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:08.182287 11455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:08.182214 11458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:08.183056 11445 server_base.cc:1034] running on GCE node
I20260812 07:04:08.183737 11445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:08.184775 11445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:08.186005 11445 hybrid_clock.cc:648] HybridClock initialized: now 1786518248185926 us; error 70 us; skew 500 ppm
I20260812 07:04:08.186364 11445 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:04:08 dist-test-slave-69rg krb5kdc[11146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518248, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:08.194468 11445 init.cc:377] Logged in from keytab as kudu/127.6.61.2@KRBTEST.COM (short username kudu)
I20260812 07:04:08.195787 11445 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:08.195885 11445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:08.196161 11445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:08.199277 11445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-1/data/instance:
uuid: "15704e1ce46548f5aa2bb7ed16ed5f6c"
format_stamp: "Formatted at 2026-08-12 07:04:08 on dist-test-slave-69rg"
server_key: "0ac90e2665bf727e715a36536bbdb9cb"
server_key_iv: "75d49c17b8bd3e334fdd991b2846191a"
server_key_version: "encryptionkey@0"
I20260812 07:04:08.200106 11445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance:
uuid: "15704e1ce46548f5aa2bb7ed16ed5f6c"
format_stamp: "Formatted at 2026-08-12 07:04:08 on dist-test-slave-69rg"
server_key: "0ac90e2665bf727e715a36536bbdb9cb"
server_key_iv: "75d49c17b8bd3e334fdd991b2846191a"
server_key_version: "encryptionkey@0"
I20260812 07:04:08.205478 11445 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 07:04:08.209095 11465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:08.210791 11445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:04:08.210937 11445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-1/wal
uuid: "15704e1ce46548f5aa2bb7ed16ed5f6c"
format_stamp: "Formatted at 2026-08-12 07:04:08 on dist-test-slave-69rg"
server_key: "0ac90e2665bf727e715a36536bbdb9cb"
server_key_iv: "75d49c17b8bd3e334fdd991b2846191a"
server_key_version: "encryptionkey@0"
I20260812 07:04:08.211127 11445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:08.236107 11445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:08.241179 11445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:08.241469 11445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:08.242588 11445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:08.244056 11445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:08.244154 11445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:08.244233 11445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:08.244280 11445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:08.282596 11445 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.2:42701
I20260812 07:04:08.282706 11600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.2:42701 every 8 connection(s)
I20260812 07:04:08.283514 11445 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
I20260812 07:04:08.292450 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 11445
I20260812 07:04:08.292675 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance
I20260812 07:04:08.293264 6388 external_mini_cluster.cc:1428] Setting key 20e3240c4f9558545b701c79419793e1
Aug 12 07:04:08 dist-test-slave-69rg krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518248, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:04:08.305918 11605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:08.286022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:54845 (local address 127.6.61.62:35795)
0812 07:04:08.286480 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:08.286485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:08.287372 (+ 887us) server_negotiation.cc:407] Connection header received
0812 07:04:08.289203 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:08.289209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:08.289311 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:08.289437 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:08.291047 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:08.291772 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:08.294118 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:08.294754 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:08.294896 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:08.298584 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:08.298593 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:08.298601 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:08.298646 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:08.302366 (+ 3720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:08.303232 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:08.303241 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:08.303246 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:08.303430 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:08.303914 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:08.303920 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:08.303925 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:08.304212 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:08.304335 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:08.305398 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0812 07:04:08.305576 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":251,"threads_started":1}
I20260812 07:04:08.307456 11607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:08.286473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:35795 (local address 127.6.61.2:54845)
0812 07:04:08.287199 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:08.287235 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:08.288927 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:08.289670 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:08.289678 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:08.290292 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:08.290860 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:08.290878 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:08.291942 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:08.291947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:08.293970 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:08.293983 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:08.294965 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:08.294970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:08.295163 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:08.296002 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:08.296026 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:08.298320 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:08.302573 (+ 4253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:08.302581 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:08.302595 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:08.303078 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:08.303574 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:08.303579 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:08.303581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:08.303772 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:08.304382 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:08.304394 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:08.304912 (+ 518us) client_negotiation.cc:769] Sending connection context
0812 07:04:08.305258 (+ 346us) client_negotiation.cc:240] Negotiation successful
0812 07:04:08.305955 (+ 697us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":298,"threads_started":1}
I20260812 07:04:08.309139 11602 heartbeater.cc:347] Connected to a master server at 127.6.61.62:35795
I20260812 07:04:08.309501 11602 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:08.310281 11602 heartbeater.cc:511] Master 127.6.61.62:35795 requested a full tablet report, sending...
I20260812 07:04:08.311952 11200 ts_manager.cc:194] Registered new tserver with Master: 15704e1ce46548f5aa2bb7ed16ed5f6c (127.6.61.2:42701)
I20260812 07:04:08.312896 11200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.2@KRBTEST.COM'} at 127.6.61.2:54845
WARNING: no policy specified for kudu/127.6.61.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
I20260812 07:04:08.350837 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.3:0
--local_ip_for_outbound_sockets=127.6.61.3
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:35795
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:34203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:08.696784 11610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:08.697281 11610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:08.697474 11610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:08.704967 11610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:08.705135 11610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:08.705335 11610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.3
I20260812 07:04:08.716261 11610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:34203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:35795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.3
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:08.717670 11610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:08.719105 11610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:08.735502 11617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:08.736394 11619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:08.736409 11616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:08.740270 11610 server_base.cc:1034] running on GCE node
I20260812 07:04:08.740959 11610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:08.742178 11610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:08.743474 11610 hybrid_clock.cc:648] HybridClock initialized: now 1786518248743406 us; error 66 us; skew 500 ppm
I20260812 07:04:08.743893 11610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:04:08 dist-test-slave-69rg krb5kdc[11146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518248, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:08.747952 11610 init.cc:377] Logged in from keytab as kudu/127.6.61.3@KRBTEST.COM (short username kudu)
I20260812 07:04:08.749543 11610 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:08.749707 11610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:08.750157 11610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:08.753486 11610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-2/data/instance:
uuid: "bd262b31ebda4b7db3f88a82c1e52763"
format_stamp: "Formatted at 2026-08-12 07:04:08 on dist-test-slave-69rg"
server_key: "44a12c1560aa3a480c9b9c3618da7967"
server_key_iv: "4a506075dfd6e3935dc7c261ad8de465"
server_key_version: "encryptionkey@0"
I20260812 07:04:08.754493 11610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance:
uuid: "bd262b31ebda4b7db3f88a82c1e52763"
format_stamp: "Formatted at 2026-08-12 07:04:08 on dist-test-slave-69rg"
server_key: "44a12c1560aa3a480c9b9c3618da7967"
server_key_iv: "4a506075dfd6e3935dc7c261ad8de465"
server_key_version: "encryptionkey@0"
I20260812 07:04:08.760506 11610 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260812 07:04:08.765244 11627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:08.767010 11610 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 07:04:08.767246 11610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-2/wal
uuid: "bd262b31ebda4b7db3f88a82c1e52763"
format_stamp: "Formatted at 2026-08-12 07:04:08 on dist-test-slave-69rg"
server_key: "44a12c1560aa3a480c9b9c3618da7967"
server_key_iv: "4a506075dfd6e3935dc7c261ad8de465"
server_key_version: "encryptionkey@0"
I20260812 07:04:08.767520 11610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:08.767942 11436 heartbeater.cc:503] Master 127.6.61.62:35795 was elected leader, sending a full tablet report...
I20260812 07:04:08.790362 11610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:08.796041 11610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:08.796437 11610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:08.797696 11610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:08.799379 11610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:08.799515 11610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:08.799660 11610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:08.799759 11610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:08.843106 11610 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.3:37829
I20260812 07:04:08.843165 11754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.3:37829 every 8 connection(s)
I20260812 07:04:08.844411 11610 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
I20260812 07:04:08.847152 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 11610
I20260812 07:04:08.847467 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance
I20260812 07:04:08.848093 6388 external_mini_cluster.cc:1428] Setting key 6e8b063f4a80106226b1b61c32f0534d
Aug 12 07:04:08 dist-test-slave-69rg krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518248, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:04:08.868268 11758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:08.847569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:60481 (local address 127.6.61.62:35795)
0812 07:04:08.848131 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:08.848138 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:08.849458 (+ 1320us) server_negotiation.cc:407] Connection header received
0812 07:04:08.851967 (+ 2509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:08.851973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:08.852112 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:08.852238 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:08.853940 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:08.854744 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:08.856471 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:08.856967 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:08.857114 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:08.861026 (+ 3912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:08.861032 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:08.861035 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:08.861071 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:08.864702 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:08.865633 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:08.865638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:08.865643 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:08.865778 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:08.866335 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:08.866340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:08.866344 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:08.866642 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:08.866828 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:08.867586 (+ 758us) server_negotiation.cc:299] Negotiation successful
0812 07:04:08.867848 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":270,"threads_started":1}
I20260812 07:04:08.869268 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:08.848401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:35795 (local address 127.6.61.3:60481)
0812 07:04:08.849230 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:08.849281 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:08.851671 (+ 2390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:08.852523 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:08.852532 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:08.853092 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:08.853720 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:08.853740 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:08.854937 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:08.854942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:08.856299 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:08.856321 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:08.857229 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:08.857237 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:08.857478 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:08.858279 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:08.858312 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:08.860778 (+ 2466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:08.864866 (+ 4088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:08.864880 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:08.864901 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:08.865478 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:08.865931 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:08.865936 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:08.865940 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:08.866196 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:08.866797 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:08.866806 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:08.867185 (+ 379us) client_negotiation.cc:769] Sending connection context
0812 07:04:08.867471 (+ 286us) client_negotiation.cc:240] Negotiation successful
0812 07:04:08.868068 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":387,"threads_started":1}
I20260812 07:04:08.871770 11756 heartbeater.cc:347] Connected to a master server at 127.6.61.62:35795
I20260812 07:04:08.872144 11756 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:08.872921 11756 heartbeater.cc:511] Master 127.6.61.62:35795 requested a full tablet report, sending...
I20260812 07:04:08.874672 11200 ts_manager.cc:194] Registered new tserver with Master: bd262b31ebda4b7db3f88a82c1e52763 (127.6.61.3:37829)
I20260812 07:04:08.875541 11200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.3@KRBTEST.COM'} at 127.6.61.3:60481
I20260812 07:04:08.877429 6388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 07:04:08.897516 11758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:08.883210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59660 (local address 127.6.61.62:35795)
0812 07:04:08.883623 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:08.883631 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:08.883983 (+ 352us) server_negotiation.cc:407] Connection header received
0812 07:04:08.884359 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:08.884364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:08.884441 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:08.884558 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:08.886296 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:08.887036 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:08.888326 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:08.888831 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:08.888984 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:08.891650 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:08.891656 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:08.891659 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:08.891696 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:08.894678 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:08.895432 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:08.895444 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:08.895448 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:08.895575 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:08.895970 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:08.895976 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:08.895979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:08.896336 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:08.896602 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:08.896940 (+ 338us) server_negotiation.cc:299] Negotiation successful
0812 07:04:08.897186 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260812 07:04:08.902813 11200 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:04:08.911105 11200 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:04:08.940213 11773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:08.925639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:37597 (local address 127.0.0.1:44564)
0812 07:04:08.926717 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:08.926776 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:08.927012 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:08.931471 (+ 4459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:08.931475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:08.931499 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:08.932246 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:08.932266 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:08.934600 (+ 2334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:08.934607 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:08.936921 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:08.936934 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:08.939227 (+ 2293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:08.939237 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:08.939444 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:08.939482 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 07:04:08.939558 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 07:04:08.939727 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"mutex_wait_us":85,"thread_start_us":1463,"threads_started":1}
I20260812 07:04:08.940203 11778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:08.926637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55198 (local address 127.6.61.3:37829)
0812 07:04:08.928096 (+ 1459us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:08.928111 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:08.929548 (+ 1437us) server_negotiation.cc:407] Connection header received
0812 07:04:08.929999 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:08.930013 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:08.930404 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:08.930656 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:08.932105 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:08.933902 (+ 1797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:08.936790 (+ 2888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:08.938754 (+ 1964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:08.938919 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:08.939220 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:08.939518 (+ 298us) server_negotiation.cc:299] Negotiation successful
0812 07:04:08.939732 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1213,"thread_start_us":1024,"threads_started":1}
I20260812 07:04:08.940820 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:08.925880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44564 (local address 127.6.61.1:37597)
0812 07:04:08.930839 (+ 4959us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:08.930851 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:08.930886 (+ 35us) server_negotiation.cc:407] Connection header received
0812 07:04:08.930989 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:08.930997 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:08.931226 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:08.931463 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:08.932517 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:08.934463 (+ 1946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:08.937185 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:08.938937 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:08.939929 (+ 992us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:08.940182 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:08.940280 (+ 98us) server_negotiation.cc:299] Negotiation successful
0812 07:04:08.940493 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4723,"thread_start_us":4534,"threads_started":1}
I20260812 07:04:08.941057 11777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:08.925782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:37829 (local address 127.0.0.1:55198)
0812 07:04:08.929393 (+ 3611us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:08.929419 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:08.929701 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:08.931098 (+ 1397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:08.931104 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:08.931139 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:08.931839 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:08.931856 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:08.934048 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:08.934054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:08.936519 (+ 2465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:08.936536 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:08.938987 (+ 2451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:08.938994 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:08.939224 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:08.939301 (+ 77us) client_negotiation.cc:769] Sending connection context
0812 07:04:08.939444 (+ 143us) client_negotiation.cc:240] Negotiation successful
0812 07:04:08.939689 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3399,"mutex_wait_us":140,"thread_start_us":4984,"threads_started":1}
I20260812 07:04:08.941859 11779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:08.926917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49880 (local address 127.6.61.2:42701)
0812 07:04:08.928747 (+ 1830us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:08.928760 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:08.931229 (+ 2469us) server_negotiation.cc:407] Connection header received
0812 07:04:08.931904 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:08.931915 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:08.932159 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:08.932379 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:08.933316 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:08.935020 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:08.938106 (+ 3086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:08.939193 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:08.939322 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:08.939520 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:08.941287 (+ 1767us) server_negotiation.cc:299] Negotiation successful
0812 07:04:08.941491 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1603,"thread_start_us":1430,"threads_started":1}
I20260812 07:04:08.941992 11774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:08.926254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:42701 (local address 127.0.0.1:49880)
0812 07:04:08.931081 (+ 4827us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:08.931098 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:08.931236 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:08.932665 (+ 1429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:08.932674 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:08.932703 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:08.933128 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:08.933138 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:08.935184 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:08.935189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:08.937931 (+ 2742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:08.937944 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:08.939676 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:08.939682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:08.940025 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:08.940056 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 07:04:08.940174 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 07:04:08.940322 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4694,"mutex_wait_us":1,"thread_start_us":2724,"threads_started":1}
I20260812 07:04:08.944916 11363 tablet_service.cc:1467] Processing CreateTablet for tablet cf1b99c1cb834965a66f38248b50e958 (DEFAULT_TABLE table=test-table [id=e90088bf2b6c4b888e4a2ddb22434ff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:04:08.945768 11682 tablet_service.cc:1467] Processing CreateTablet for tablet cf1b99c1cb834965a66f38248b50e958 (DEFAULT_TABLE table=test-table [id=e90088bf2b6c4b888e4a2ddb22434ff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:04:08.946293 11521 tablet_service.cc:1467] Processing CreateTablet for tablet cf1b99c1cb834965a66f38248b50e958 (DEFAULT_TABLE table=test-table [id=e90088bf2b6c4b888e4a2ddb22434ff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:04:08.946955 11363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf1b99c1cb834965a66f38248b50e958. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:08.947412 11682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf1b99c1cb834965a66f38248b50e958. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:08.947569 11521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf1b99c1cb834965a66f38248b50e958. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:08.958846 11781 tablet_bootstrap.cc:492] T cf1b99c1cb834965a66f38248b50e958 P e8dc27683e6c4188bbe0ec4c397b8796: Bootstrap starting.
I20260812 07:04:08.962837 11782 tablet_bootstrap.cc:492] T cf1b99c1cb834965a66f38248b50e958 P 15704e1ce46548f5aa2bb7ed16ed5f6c: Bootstrap starting.
I20260812 07:04:08.963841 11781 tablet_bootstrap.cc:654] T cf1b99c1cb834965a66f38248b50e958 P e8dc27683e6c4188bbe0ec4c397b8796: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:08.965682 11781 log.cc:824] T cf1b99c1cb834965a66f38248b50e958 P e8dc27683e6c4188bbe0ec4c397b8796: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:08.966795 11783 tablet_bootstrap.cc:492] T cf1b99c1cb834965a66f38248b50e958 P bd262b31ebda4b7db3f88a82c1e52763: Bootstrap starting.
I20260812 07:04:08.967684 11782 tablet_bootstrap.cc:654] T cf1b99c1cb834965a66f38248b50e958 P 15704e1ce46548f5aa2bb7ed16ed5f6c: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:08.969375 11782 log.cc:824] T cf1b99c1cb834965a66f38248b50e958 P 15704e1ce46548f5aa2bb7ed16ed5f6c: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:08.971040 11783 tablet_bootstrap.cc:654] T cf1b99c1cb834965a66f38248b50e958 P bd262b31ebda4b7db3f88a82c1e52763: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:08.972041 11781 tablet_bootstrap.cc:492] T cf1b99c1cb834965a66f38248b50e958 P e8dc27683e6c4188bbe0ec4c397b8796: No bootstrap required, opened a new log
I20260812 07:04:08.972415 11783 log.cc:824] T cf1b99c1cb834965a66f38248b50e958 P bd262b31ebda4b7db3f88a82c1e52763: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:08.972473 11781 ts_tablet_manager.cc:1397] T cf1b99c1cb834965a66f38248b50e958 P e8dc27683e6c4188bbe0ec4c397b8796: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.004s
I20260812 07:04:08.973268 11782 tablet_bootstrap.cc:492] T cf1b99c1cb834965a66f38248b50e958 P 15704e1ce46548f5aa2bb7ed16ed5f6c: No bootstrap required, opened a new log
I20260812 07:04:08.973563 11782 ts_tablet_manager.cc:1397] T cf1b99c1cb834965a66f38248b50e958 P 15704e1ce46548f5aa2bb7ed16ed5f6c: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260812 07:04:08.975211 11783 tablet_bootstrap.cc:492] T cf1b99c1cb834965a66f38248b50e958 P bd262b31ebda4b7db3f88a82c1e52763: No bootstrap required, opened a new log
I20260812 07:04:08.975476 11783 ts_tablet_manager.cc:1397] T cf1b99c1cb834965a66f38248b50e958 P bd262b31ebda4b7db3f88a82c1e52763: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260812 07:04:08.983501 11783 raft_consensus.cc:348] T cf1b99c1cb834965a66f38248b50e958 P bd262b31ebda4b7db3f88a82c1e52763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8dc27683e6c4188bbe0ec4c397b8796" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 37597 } } peers { permanent_uuid: "bd262b31ebda4b7db3f88a82c1e52763" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 37829 } } peers { permanent_uuid: "15704e1ce46548f5aa2bb7ed16ed5f6c" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 42701 } }
I20260812 07:04:08.984053 11783 raft_consensus.cc:374] T cf1b99c1cb834965a66f38248b50e958 P bd262b31ebda4b7db3f88a82c1e52763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:08.984172 11783 raft_consensus.cc:729] T cf1b99c1cb834965a66f38248b50e958 P bd262b31ebda4b7db3f88a82c1e52763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd262b31ebda4b7db3f88a82c1e52763, State: Initialized, Role: FOLLOWER
I20260812 07:04:08.984599 11781 raft_consensus.cc:348] T cf1b99c1cb834965a66f38248b50e958 P e8dc27683e6c4188bbe0ec4c397b8796 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8dc27683e6c4188bbe0ec4c397b8796" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 37597 } } peers { permanent_uuid: "bd262b31ebda4b7db3f88a82c1e52763" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 37829 } } peers { permanent_uuid: "15704e1ce46548f5aa2bb7ed16ed5f6c" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 42701 } }
I20260812 07:04:08.984599 11782 raft_consensus.cc:348] T cf1b99c1cb834965a66f38248b50e958 P 15704e1ce46548f5aa2bb7ed16ed5f6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8dc27683e6c4188bbe0ec4c397b8796" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 37597 } } peers { permanent_uuid: "bd262b31ebda4b7db3f88a82c1e52763" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 37829 } } peers { permanent_uuid: "15704e1ce46548f5aa2bb7ed16ed5f6c" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 42701 } }
I20260812 07:04:08.985036 11781 raft_consensus.cc:374] T cf1b99c1cb834965a66f38248b50e958 P e8dc27683e6c4188bbe0ec4c397b8796 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:08.985153 11781 raft_consensus.cc:729] T cf1b99c1cb834965a66f38248b50e958 P e8dc27683e6c4188bbe0ec4c397b8796 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8dc27683e6c4188bbe0ec4c397b8796, State: Initialized, Role: FOLLOWER
I20260812 07:04:08.985177 11782 raft_consensus.cc:374] T cf1b99c1cb834965a66f38248b50e958 P 15704e1ce46548f5aa2bb7ed16ed5f6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:08.985013 11783 consensus_queue.cc:260] T cf1b99c1cb834965a66f38248b50e958 P bd262b31ebda4b7db3f88a82c1e52763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8dc27683e6c4188bbe0ec4c397b8796" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 37597 } } peers { permanent_uuid: "bd262b31ebda4b7db3f88a82c1e52763" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 37829 } } peers { permanent_uuid: "15704e1ce46548f5aa2bb7ed16ed5f6c" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 42701 } }
I20260812 07:04:08.985270 11782 raft_consensus.cc:729] T cf1b99c1cb834965a66f38248b50e958 P 15704e1ce46548f5aa2bb7ed16ed5f6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15704e1ce46548f5aa2bb7ed16ed5f6c, State: Initialized, Role: FOLLOWER
I20260812 07:04:08.986022 11782 consensus_queue.cc:260] T cf1b99c1cb834965a66f38248b50e958 P 15704e1ce46548f5aa2bb7ed16ed5f6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8dc27683e6c4188bbe0ec4c397b8796" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 37597 } } peers { permanent_uuid: "bd262b31ebda4b7db3f88a82c1e52763" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 37829 } } peers { permanent_uuid: "15704e1ce46548f5aa2bb7ed16ed5f6c" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 42701 } }
I20260812 07:04:08.986022 11781 consensus_queue.cc:260] T cf1b99c1cb834965a66f38248b50e958 P e8dc27683e6c4188bbe0ec4c397b8796 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8dc27683e6c4188bbe0ec4c397b8796" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 37597 } } peers { permanent_uuid: "bd262b31ebda4b7db3f88a82c1e52763" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 37829 } } peers { permanent_uuid: "15704e1ce46548f5aa2bb7ed16ed5f6c" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 42701 } }
I20260812 07:04:08.986509 11756 heartbeater.cc:503] Master 127.6.61.62:35795 was elected leader, sending a full tablet report...
I20260812 07:04:08.988276 11781 ts_tablet_manager.cc:1428] T cf1b99c1cb834965a66f38248b50e958 P e8dc27683e6c4188bbe0ec4c397b8796: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
I20260812 07:04:08.990844 11783 ts_tablet_manager.cc:1428] T cf1b99c1cb834965a66f38248b50e958 P bd262b31ebda4b7db3f88a82c1e52763: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260812 07:04:08.991312 11602 heartbeater.cc:503] Master 127.6.61.62:35795 was elected leader, sending a full tablet report...
I20260812 07:04:08.991998 11782 ts_tablet_manager.cc:1428] T cf1b99c1cb834965a66f38248b50e958 P 15704e1ce46548f5aa2bb7ed16ed5f6c: Time spent starting tablet: real 0.018s user 0.011s sys 0.008s
W20260812 07:04:09.035593 11604 tablet.cc:2369] T cf1b99c1cb834965a66f38248b50e958 P 15704e1ce46548f5aa2bb7ed16ed5f6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 07:04:09.096243 11757 tablet.cc:2369] T cf1b99c1cb834965a66f38248b50e958 P bd262b31ebda4b7db3f88a82c1e52763: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:04:09.125034 11789 raft_consensus.cc:482] T cf1b99c1cb834965a66f38248b50e958 P 15704e1ce46548f5aa2bb7ed16ed5f6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:04:09.125372 11789 raft_consensus.cc:504] T cf1b99c1cb834965a66f38248b50e958 P 15704e1ce46548f5aa2bb7ed16ed5f6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8dc27683e6c4188bbe0ec4c397b8796" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 37597 } } peers { permanent_uuid: "bd262b31ebda4b7db3f88a82c1e52763" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 37829 } } peers { permanent_uuid: "15704e1ce46548f5aa2bb7ed16ed5f6c" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 42701 } }
I20260812 07:04:09.127230 11789 leader_election.cc:288] T cf1b99c1cb834965a66f38248b50e958 P 15704e1ce46548f5aa2bb7ed16ed5f6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8dc27683e6c4188bbe0ec4c397b8796 (127.6.61.1:37597), bd262b31ebda4b7db3f88a82c1e52763 (127.6.61.3:37829)
I20260812 07:04:09.136971 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:09.127942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:60581 (local address 127.6.61.1:37597)
0812 07:04:09.128467 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:09.128475 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:09.128853 (+ 378us) server_negotiation.cc:407] Connection header received
0812 07:04:09.129005 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:09.129011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:09.129095 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:09.129187 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:09.130044 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:09.131295 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:09.134396 (+ 3101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:09.135685 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:09.135841 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:09.135958 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:09.136541 (+ 583us) server_negotiation.cc:299] Negotiation successful
0812 07:04:09.136688 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319}
I20260812 07:04:09.137182 11778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:09.128199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:34557 (local address 127.6.61.3:37829)
0812 07:04:09.128643 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:09.128649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:09.128672 (+ 23us) server_negotiation.cc:407] Connection header received
0812 07:04:09.128891 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:09.128897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:09.129004 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:09.129118 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:09.130044 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:09.131295 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:09.134360 (+ 3065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:09.135685 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:09.135841 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:09.135957 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:09.136538 (+ 581us) server_negotiation.cc:299] Negotiation successful
0812 07:04:09.136688 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300}
I20260812 07:04:09.137192 11791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:09.127559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:37597 (local address 127.6.61.2:60581)
0812 07:04:09.128747 (+ 1188us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:09.128766 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:09.128881 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:09.129212 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:09.129215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:09.129236 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:09.129793 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:09.129801 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:09.131491 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:09.131495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:09.134120 (+ 2625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:09.134139 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:09.136027 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:09.136032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:09.136285 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:09.136346 (+ 61us) client_negotiation.cc:769] Sending connection context
0812 07:04:09.136460 (+ 114us) client_negotiation.cc:240] Negotiation successful
0812 07:04:09.136629 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1071,"mutex_wait_us":101,"thread_start_us":866,"threads_started":1}
I20260812 07:04:09.137256 11790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:09.127511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:37829 (local address 127.6.61.2:34557)
0812 07:04:09.128464 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:09.128492 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:09.128691 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:09.129142 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:09.129167 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:09.129199 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:09.129785 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:09.129801 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:09.131485 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:09.131491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:09.134120 (+ 2629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:09.134139 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:09.136024 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:09.136032 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:09.136281 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:09.136346 (+ 65us) client_negotiation.cc:769] Sending connection context
0812 07:04:09.136460 (+ 114us) client_negotiation.cc:240] Negotiation successful
0812 07:04:09.136661 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":256,"thread_start_us":571,"threads_started":1}
I20260812 07:04:09.138616 11386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf1b99c1cb834965a66f38248b50e958" candidate_uuid: "15704e1ce46548f5aa2bb7ed16ed5f6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8dc27683e6c4188bbe0ec4c397b8796" is_pre_election: true
I20260812 07:04:09.138746 11707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf1b99c1cb834965a66f38248b50e958" candidate_uuid: "15704e1ce46548f5aa2bb7ed16ed5f6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd262b31ebda4b7db3f88a82c1e52763" is_pre_election: true
I20260812 07:04:09.139213 11386 raft_consensus.cc:2454] T cf1b99c1cb834965a66f38248b50e958 P e8dc27683e6c4188bbe0ec4c397b8796 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15704e1ce46548f5aa2bb7ed16ed5f6c in term 0.
I20260812 07:04:09.139278 11707 raft_consensus.cc:2454] T cf1b99c1cb834965a66f38248b50e958 P bd262b31ebda4b7db3f88a82c1e52763 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15704e1ce46548f5aa2bb7ed16ed5f6c in term 0.
I20260812 07:04:09.140043 11466 leader_election.cc:302] T cf1b99c1cb834965a66f38248b50e958 P 15704e1ce46548f5aa2bb7ed16ed5f6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 15704e1ce46548f5aa2bb7ed16ed5f6c, bd262b31ebda4b7db3f88a82c1e52763, e8dc27683e6c4188bbe0ec4c397b8796; no voters:
I20260812 07:04:09.140558 11789 raft_consensus.cc:2781] T cf1b99c1cb834965a66f38248b50e958 P 15704e1ce46548f5aa2bb7ed16ed5f6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:04:09.140765 11789 raft_consensus.cc:482] T cf1b99c1cb834965a66f38248b50e958 P 15704e1ce46548f5aa2bb7ed16ed5f6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:04:09.140895 11789 raft_consensus.cc:3037] T cf1b99c1cb834965a66f38248b50e958 P 15704e1ce46548f5aa2bb7ed16ed5f6c [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:09.143299 11789 raft_consensus.cc:504] T cf1b99c1cb834965a66f38248b50e958 P 15704e1ce46548f5aa2bb7ed16ed5f6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8dc27683e6c4188bbe0ec4c397b8796" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 37597 } } peers { permanent_uuid: "bd262b31ebda4b7db3f88a82c1e52763" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 37829 } } peers { permanent_uuid: "15704e1ce46548f5aa2bb7ed16ed5f6c" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 42701 } }
I20260812 07:04:09.143968 11789 leader_election.cc:288] T cf1b99c1cb834965a66f38248b50e958 P 15704e1ce46548f5aa2bb7ed16ed5f6c [CANDIDATE]: Term 1 election: Requested vote from peers e8dc27683e6c4188bbe0ec4c397b8796 (127.6.61.1:37597), bd262b31ebda4b7db3f88a82c1e52763 (127.6.61.3:37829)
I20260812 07:04:09.144587 11707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf1b99c1cb834965a66f38248b50e958" candidate_uuid: "15704e1ce46548f5aa2bb7ed16ed5f6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd262b31ebda4b7db3f88a82c1e52763"
I20260812 07:04:09.144574 11386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf1b99c1cb834965a66f38248b50e958" candidate_uuid: "15704e1ce46548f5aa2bb7ed16ed5f6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8dc27683e6c4188bbe0ec4c397b8796"
I20260812 07:04:09.144800 11707 raft_consensus.cc:3037] T cf1b99c1cb834965a66f38248b50e958 P bd262b31ebda4b7db3f88a82c1e52763 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:09.144824 11386 raft_consensus.cc:3037] T cf1b99c1cb834965a66f38248b50e958 P e8dc27683e6c4188bbe0ec4c397b8796 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:09.147239 11707 raft_consensus.cc:2454] T cf1b99c1cb834965a66f38248b50e958 P bd262b31ebda4b7db3f88a82c1e52763 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15704e1ce46548f5aa2bb7ed16ed5f6c in term 1.
I20260812 07:04:09.147830 11468 leader_election.cc:302] T cf1b99c1cb834965a66f38248b50e958 P 15704e1ce46548f5aa2bb7ed16ed5f6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15704e1ce46548f5aa2bb7ed16ed5f6c, bd262b31ebda4b7db3f88a82c1e52763; no voters:
I20260812 07:04:09.147876 11386 raft_consensus.cc:2454] T cf1b99c1cb834965a66f38248b50e958 P e8dc27683e6c4188bbe0ec4c397b8796 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15704e1ce46548f5aa2bb7ed16ed5f6c in term 1.
I20260812 07:04:09.148185 11789 raft_consensus.cc:2781] T cf1b99c1cb834965a66f38248b50e958 P 15704e1ce46548f5aa2bb7ed16ed5f6c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:04:09.148824 11789 raft_consensus.cc:686] T cf1b99c1cb834965a66f38248b50e958 P 15704e1ce46548f5aa2bb7ed16ed5f6c [term 1 LEADER]: Becoming Leader. State: Replica: 15704e1ce46548f5aa2bb7ed16ed5f6c, State: Running, Role: LEADER
I20260812 07:04:09.149386 11789 consensus_queue.cc:237] T cf1b99c1cb834965a66f38248b50e958 P 15704e1ce46548f5aa2bb7ed16ed5f6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8dc27683e6c4188bbe0ec4c397b8796" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 37597 } } peers { permanent_uuid: "bd262b31ebda4b7db3f88a82c1e52763" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 37829 } } peers { permanent_uuid: "15704e1ce46548f5aa2bb7ed16ed5f6c" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 42701 } }
I20260812 07:04:09.154799 11197 catalog_manager.cc:5495] T cf1b99c1cb834965a66f38248b50e958 P 15704e1ce46548f5aa2bb7ed16ed5f6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 15704e1ce46548f5aa2bb7ed16ed5f6c (127.6.61.2). New cstate: current_term: 1 leader_uuid: "15704e1ce46548f5aa2bb7ed16ed5f6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8dc27683e6c4188bbe0ec4c397b8796" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 37597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd262b31ebda4b7db3f88a82c1e52763" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 37829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15704e1ce46548f5aa2bb7ed16ed5f6c" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 42701 } health_report { overall_health: HEALTHY } } }
I20260812 07:04:09.184191 11779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:09.176906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35356 (local address 127.6.61.2:42701)
0812 07:04:09.177096 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:09.177100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:09.177119 (+ 19us) server_negotiation.cc:407] Connection header received
0812 07:04:09.177338 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:09.177342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:09.177433 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:09.177562 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:04:09.178140 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:09.179036 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:09.180922 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:09.181377 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:09.181495 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:09.182452 (+ 957us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:04:09.183274 (+ 822us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:04:09.183489 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:09.183795 (+ 306us) server_negotiation.cc:299] Negotiation successful
0812 07:04:09.183900 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 07:04:09.194368 11707 raft_consensus.cc:1261] T cf1b99c1cb834965a66f38248b50e958 P bd262b31ebda4b7db3f88a82c1e52763 [term 1 FOLLOWER]: Refusing update from remote peer 15704e1ce46548f5aa2bb7ed16ed5f6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:04:09.194370 11386 raft_consensus.cc:1261] T cf1b99c1cb834965a66f38248b50e958 P e8dc27683e6c4188bbe0ec4c397b8796 [term 1 FOLLOWER]: Refusing update from remote peer 15704e1ce46548f5aa2bb7ed16ed5f6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:04:09.195429 11793 consensus_queue.cc:1035] T cf1b99c1cb834965a66f38248b50e958 P 15704e1ce46548f5aa2bb7ed16ed5f6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd262b31ebda4b7db3f88a82c1e52763" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:04:09.195802 11789 consensus_queue.cc:1035] T cf1b99c1cb834965a66f38248b50e958 P 15704e1ce46548f5aa2bb7ed16ed5f6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8dc27683e6c4188bbe0ec4c397b8796" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 37597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:04:09.205883 11798 mvcc.cc:205] Tried to move back new op lower bound from 7317578748688105472 to 7317578748520128512. Current Snapshot: MvccSnapshot[applied={T|T < 7317578748688105472}]
I20260812 07:04:09.208420 11803 mvcc.cc:205] Tried to move back new op lower bound from 7317578748688105472 to 7317578748520128512. Current Snapshot: MvccSnapshot[applied={T|T < 7317578748688105472}]
I20260812 07:04:09.213143 11802 mvcc.cc:205] Tried to move back new op lower bound from 7317578748688105472 to 7317578748520128512. Current Snapshot: MvccSnapshot[applied={T|T < 7317578748688105472}]
I20260812 07:04:09.222891 11198 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:04:09.223191 11198 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:04:09.226969 11198 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 206b848b010543aa9e5e4b597a19285c: Sending DeleteTablet for 3 replicas of tablet cf1b99c1cb834965a66f38248b50e958
I20260812 07:04:09.228130 11682 tablet_service.cc:1514] Processing DeleteTablet for tablet cf1b99c1cb834965a66f38248b50e958 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:04:09 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:55198
I20260812 07:04:09.228171 11363 tablet_service.cc:1514] Processing DeleteTablet for tablet cf1b99c1cb834965a66f38248b50e958 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:04:09 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:44564
I20260812 07:04:09.228858 11521 tablet_service.cc:1514] Processing DeleteTablet for tablet cf1b99c1cb834965a66f38248b50e958 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:04:09 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:49880
I20260812 07:04:09.229439 11818 tablet_replica.cc:330] T cf1b99c1cb834965a66f38248b50e958 P e8dc27683e6c4188bbe0ec4c397b8796: stopping tablet replica
I20260812 07:04:09.230093 11818 raft_consensus.cc:2229] T cf1b99c1cb834965a66f38248b50e958 P e8dc27683e6c4188bbe0ec4c397b8796 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:04:09.230806 11818 raft_consensus.cc:2258] T cf1b99c1cb834965a66f38248b50e958 P e8dc27683e6c4188bbe0ec4c397b8796 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:04:09.234346 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 11277
I20260812 07:04:09.240137 11817 tablet_replica.cc:330] T cf1b99c1cb834965a66f38248b50e958 P bd262b31ebda4b7db3f88a82c1e52763: stopping tablet replica
I20260812 07:04:09.240836 11817 raft_consensus.cc:2229] T cf1b99c1cb834965a66f38248b50e958 P bd262b31ebda4b7db3f88a82c1e52763 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:04:09.241487 11817 raft_consensus.cc:2258] T cf1b99c1cb834965a66f38248b50e958 P bd262b31ebda4b7db3f88a82c1e52763 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:04:09.242658 11818 ts_tablet_manager.cc:1905] T cf1b99c1cb834965a66f38248b50e958 P e8dc27683e6c4188bbe0ec4c397b8796: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:04:09.244195 11817 ts_tablet_manager.cc:1905] T cf1b99c1cb834965a66f38248b50e958 P bd262b31ebda4b7db3f88a82c1e52763: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:04:09.251353 11818 ts_tablet_manager.cc:1918] T cf1b99c1cb834965a66f38248b50e958 P e8dc27683e6c4188bbe0ec4c397b8796: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:04:09.251706 11818 log.cc:1196] T cf1b99c1cb834965a66f38248b50e958 P e8dc27683e6c4188bbe0ec4c397b8796: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-0/wal/wals/cf1b99c1cb834965a66f38248b50e958
I20260812 07:04:09.252478 11818 ts_tablet_manager.cc:1939] T cf1b99c1cb834965a66f38248b50e958 P e8dc27683e6c4188bbe0ec4c397b8796: Deleting consensus metadata
I20260812 07:04:09.253937 11817 ts_tablet_manager.cc:1918] T cf1b99c1cb834965a66f38248b50e958 P bd262b31ebda4b7db3f88a82c1e52763: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:04:09.254215 11817 log.cc:1196] T cf1b99c1cb834965a66f38248b50e958 P bd262b31ebda4b7db3f88a82c1e52763: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-2/wal/wals/cf1b99c1cb834965a66f38248b50e958
I20260812 07:04:09.254282 11819 tablet_replica.cc:330] T cf1b99c1cb834965a66f38248b50e958 P 15704e1ce46548f5aa2bb7ed16ed5f6c: stopping tablet replica
I20260812 07:04:09.254666 11181 catalog_manager.cc:4841] TS e8dc27683e6c4188bbe0ec4c397b8796 (127.6.61.1:37597): tablet cf1b99c1cb834965a66f38248b50e958 (table test-table [id=e90088bf2b6c4b888e4a2ddb22434ff3]) successfully deleted
I20260812 07:04:09.254818 11819 raft_consensus.cc:2229] T cf1b99c1cb834965a66f38248b50e958 P 15704e1ce46548f5aa2bb7ed16ed5f6c [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:04:09.254884 11817 ts_tablet_manager.cc:1939] T cf1b99c1cb834965a66f38248b50e958 P bd262b31ebda4b7db3f88a82c1e52763: Deleting consensus metadata
I20260812 07:04:09.255527 11819 raft_consensus.cc:2258] T cf1b99c1cb834965a66f38248b50e958 P 15704e1ce46548f5aa2bb7ed16ed5f6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:04:09.256592 11182 catalog_manager.cc:4841] TS bd262b31ebda4b7db3f88a82c1e52763 (127.6.61.3:37829): tablet cf1b99c1cb834965a66f38248b50e958 (table test-table [id=e90088bf2b6c4b888e4a2ddb22434ff3]) successfully deleted
I20260812 07:04:09.257905 11819 ts_tablet_manager.cc:1905] T cf1b99c1cb834965a66f38248b50e958 P 15704e1ce46548f5aa2bb7ed16ed5f6c: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 12 07:04:09 dist-test-slave-69rg krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.1@KRBTEST.COM
I20260812 07:04:09.265697 11819 ts_tablet_manager.cc:1918] T cf1b99c1cb834965a66f38248b50e958 P 15704e1ce46548f5aa2bb7ed16ed5f6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:04:09.265975 11819 log.cc:1196] T cf1b99c1cb834965a66f38248b50e958 P 15704e1ce46548f5aa2bb7ed16ed5f6c: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518208686283-6388-0/minicluster-data/ts-1/wal/wals/cf1b99c1cb834965a66f38248b50e958
I20260812 07:04:09.266605 11819 ts_tablet_manager.cc:1939] T cf1b99c1cb834965a66f38248b50e958 P 15704e1ce46548f5aa2bb7ed16ed5f6c: Deleting consensus metadata
I20260812 07:04:09.268038 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:09.236542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51692 (local address 127.6.61.1:37597)
0812 07:04:09.236759 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:09.236768 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:09.245673 (+ 8905us) server_negotiation.cc:407] Connection header received
0812 07:04:09.245910 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:09.245918 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:09.246036 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:09.246172 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:09.248081 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:09.249308 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:09.251393 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:09.252358 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:09.252472 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:09.261300 (+ 8828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:09.261322 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:09.261337 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:09.261382 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:09.264491 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:09.265407 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:09.265416 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:09.265424 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:09.265565 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:09.266103 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:09.266108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:09.266112 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:09.266579 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:09.266879 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:09.267400 (+ 521us) server_negotiation.cc:299] Negotiation successful
0812 07:04:09.267638 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 07:04:09.268401 11184 catalog_manager.cc:4841] TS 15704e1ce46548f5aa2bb7ed16ed5f6c (127.6.61.2:42701): tablet cf1b99c1cb834965a66f38248b50e958 (table test-table [id=e90088bf2b6c4b888e4a2ddb22434ff3]) successfully deleted
I20260812 07:04:09.268792 11431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:09.377170 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 11277
I20260812 07:04:09.400619 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 11445
Aug 12 07:04:09 dist-test-slave-69rg krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.2@KRBTEST.COM
I20260812 07:04:09.417090 11779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:09.401654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35370 (local address 127.6.61.2:42701)
0812 07:04:09.401920 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:09.401931 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:09.401955 (+ 24us) server_negotiation.cc:407] Connection header received
0812 07:04:09.402061 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:09.402069 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:09.402148 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:09.402273 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:09.403557 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:09.404416 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:09.405670 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:09.406235 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:09.406382 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:09.410750 (+ 4368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:09.410762 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:09.410782 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:09.410820 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:09.414087 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:09.414965 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:09.414972 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:09.414974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:09.415111 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:09.415495 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:09.415498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:09.415500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:09.416006 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:09.416241 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:09.416539 (+ 298us) server_negotiation.cc:299] Negotiation successful
0812 07:04:09.416705 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260812 07:04:09.417971 11596 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:09.552834 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 11445
I20260812 07:04:09.577438 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 11610
Aug 12 07:04:09 dist-test-slave-69rg krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.3@KRBTEST.COM
I20260812 07:04:09.592607 11778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:09.578291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40678 (local address 127.6.61.3:37829)
0812 07:04:09.578469 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:09.578474 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:09.578554 (+ 80us) server_negotiation.cc:407] Connection header received
0812 07:04:09.578709 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:09.578713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:09.578783 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:09.578870 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:09.580096 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:09.580828 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:09.582292 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:09.582691 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:09.582799 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:09.587184 (+ 4385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:09.587198 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:09.587216 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:09.587256 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:09.590147 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:09.590751 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:09.590758 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:09.590760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:09.590898 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:09.591222 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:09.591226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:09.591228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:09.591568 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:09.591792 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:09.592116 (+ 324us) server_negotiation.cc:299] Negotiation successful
0812 07:04:09.592320 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 07:04:09.593298 11750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:09.686568 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 11610
I20260812 07:04:09.708685 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 11162
I20260812 07:04:09.709951 11239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:09.821146 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 11162
2026-08-12T07:04:09Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4659 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-12T07:04:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:04:09Z Disabled control of system clock
I20260812 07:04:09.909171 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:34537
--webserver_interface=127.6.61.62
--webserver_port=0
--builtin_ntp_servers=127.6.61.20:38557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:34537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 07:04:10.255905 11831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:10.256215 11831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:10.256304 11831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:10.263743 11831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:04:10.263856 11831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:10.263895 11831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:10.263954 11831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:04:10.263999 11831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:04:10.273368 11831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:38557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:34537
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:34537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:10.274677 11831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:10.276444 11831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:10.288542 11836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:10.288635 11837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:10.288735 11839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:10.289255 11831 server_base.cc:1034] running on GCE node
I20260812 07:04:10.290128 11831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:10.291491 11831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:10.292757 11831 hybrid_clock.cc:648] HybridClock initialized: now 1786518250292707 us; error 58 us; skew 500 ppm
I20260812 07:04:10.293169 11831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 07:04:10.295681 11831 webserver.cc:460] Webserver started at http://127.6.61.62:43347/ using document root <none> and password file <none>
I20260812 07:04:10.296402 11831 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:10.296537 11831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:10.296890 11831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:10.299980 11831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "10df99979f384f8b860d425567851c7b"
format_stamp: "Formatted at 2026-08-12 07:04:10 on dist-test-slave-69rg"
server_key: "b5c34222f7635ef76b4dae9d61ee26ff"
server_key_iv: "a6e2b6d55703023a13c9901d730fd886"
server_key_version: "encryptionkey@0"
I20260812 07:04:10.300868 11831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "10df99979f384f8b860d425567851c7b"
format_stamp: "Formatted at 2026-08-12 07:04:10 on dist-test-slave-69rg"
server_key: "b5c34222f7635ef76b4dae9d61ee26ff"
server_key_iv: "a6e2b6d55703023a13c9901d730fd886"
server_key_version: "encryptionkey@0"
I20260812 07:04:10.306294 11831 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 07:04:10.309983 11845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:10.311542 11831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:04:10.311688 11831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "10df99979f384f8b860d425567851c7b"
format_stamp: "Formatted at 2026-08-12 07:04:10 on dist-test-slave-69rg"
server_key: "b5c34222f7635ef76b4dae9d61ee26ff"
server_key_iv: "a6e2b6d55703023a13c9901d730fd886"
server_key_version: "encryptionkey@0"
I20260812 07:04:10.311877 11831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:10.317617 11831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:10.318462 11831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:10.318719 11831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:10.337044 11831 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:34537
I20260812 07:04:10.337106 11903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:34537 every 8 connection(s)
I20260812 07:04:10.338546 11831 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:04:10.341935 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 11831
I20260812 07:04:10.342133 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:04:10.342621 11904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:10.342916 6388 external_mini_cluster.cc:1428] Setting key 9fe96808dd4974dd416784b74bc40cd5
I20260812 07:04:10.351649 11908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:10.345093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46856 (local address 127.6.61.62:34537)
0812 07:04:10.346321 (+ 1228us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:10.346332 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:10.346376 (+ 44us) server_negotiation.cc:407] Connection header received
0812 07:04:10.347925 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:10.347948 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:10.348378 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:10.348814 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:10.349974 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:10.349980 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:10.349993 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:04:10.350008 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:10.350085 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:04:10.350104 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:10.350355 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:10.350501 (+ 146us) server_negotiation.cc:299] Negotiation successful
0812 07:04:10.350758 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":406,"threads_started":1}
I20260812 07:04:10.356442 11904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10df99979f384f8b860d425567851c7b: Bootstrap starting.
I20260812 07:04:10.359622 11904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10df99979f384f8b860d425567851c7b: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:10.360888 11904 log.cc:824] T 00000000000000000000000000000000 P 10df99979f384f8b860d425567851c7b: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:10.363998 11904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10df99979f384f8b860d425567851c7b: No bootstrap required, opened a new log
I20260812 07:04:10.371910 11904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10df99979f384f8b860d425567851c7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10df99979f384f8b860d425567851c7b" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34537 } }
I20260812 07:04:10.372293 11904 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10df99979f384f8b860d425567851c7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:10.372362 11904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10df99979f384f8b860d425567851c7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10df99979f384f8b860d425567851c7b, State: Initialized, Role: FOLLOWER
I20260812 07:04:10.372890 11904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10df99979f384f8b860d425567851c7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10df99979f384f8b860d425567851c7b" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34537 } }
I20260812 07:04:10.373164 11904 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10df99979f384f8b860d425567851c7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:04:10.373270 11904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10df99979f384f8b860d425567851c7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:04:10.373389 11904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10df99979f384f8b860d425567851c7b [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:10.375622 11904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10df99979f384f8b860d425567851c7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10df99979f384f8b860d425567851c7b" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34537 } }
I20260812 07:04:10.376101 11904 leader_election.cc:302] T 00000000000000000000000000000000 P 10df99979f384f8b860d425567851c7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10df99979f384f8b860d425567851c7b; no voters:
I20260812 07:04:10.376696 11904 leader_election.cc:288] T 00000000000000000000000000000000 P 10df99979f384f8b860d425567851c7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:04:10.376874 11910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10df99979f384f8b860d425567851c7b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:04:10.377696 11910 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10df99979f384f8b860d425567851c7b [term 1 LEADER]: Becoming Leader. State: Replica: 10df99979f384f8b860d425567851c7b, State: Running, Role: LEADER
I20260812 07:04:10.378504 11910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10df99979f384f8b860d425567851c7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10df99979f384f8b860d425567851c7b" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34537 } }
I20260812 07:04:10.379165 11904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10df99979f384f8b860d425567851c7b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:04:10.381628 11911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10df99979f384f8b860d425567851c7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10df99979f384f8b860d425567851c7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10df99979f384f8b860d425567851c7b" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34537 } } }
I20260812 07:04:10.382133 11911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10df99979f384f8b860d425567851c7b [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:10.381963 11913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10df99979f384f8b860d425567851c7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10df99979f384f8b860d425567851c7b. Latest consensus state: current_term: 1 leader_uuid: "10df99979f384f8b860d425567851c7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10df99979f384f8b860d425567851c7b" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34537 } } }
I20260812 07:04:10.382371 11913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10df99979f384f8b860d425567851c7b [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:10.384824 11920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:04:10.392133 11920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:04:10.402909 11920 catalog_manager.cc:1347] Generated new cluster ID: 0cb143c49b324ddb80770c6f945dfd4f
I20260812 07:04:10.403080 11920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:04:10.417225 11920 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:04:10.417729 11920 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:04:10.428642 11920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10df99979f384f8b860d425567851c7b: Generated new TSK 0
I20260812 07:04:10.429575 11920 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:04:10.440831 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.1:0
--local_ip_for_outbound_sockets=127.6.61.1
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:34537
--builtin_ntp_servers=127.6.61.20:38557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 07:04:10.811305 11936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:10.811635 11936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:10.811731 11936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:10.819958 11936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:10.820091 11936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:10.820206 11936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.1
I20260812 07:04:10.829869 11936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:38557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:34537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.1
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:10.831187 11936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:10.833042 11936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:10.848611 11941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:10.848694 11942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:10.848639 11944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:10.849651 11936 server_base.cc:1034] running on GCE node
I20260812 07:04:10.850565 11936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:10.851884 11936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:10.853343 11936 hybrid_clock.cc:648] HybridClock initialized: now 1786518250853269 us; error 65 us; skew 500 ppm
I20260812 07:04:10.853888 11936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 07:04:10.857296 11936 webserver.cc:460] Webserver started at http://127.6.61.1:35211/ using document root <none> and password file <none>
I20260812 07:04:10.858217 11936 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:10.858388 11936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:10.858773 11936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:10.862358 11936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-0/data/instance:
uuid: "8f6cc82b78e44865be6248e20b9e5b8f"
format_stamp: "Formatted at 2026-08-12 07:04:10 on dist-test-slave-69rg"
server_key: "fc630ac47a64e389e0bf7dd603c7022f"
server_key_iv: "c8d07791a9261e771d5cd2748b7f3cf0"
server_key_version: "encryptionkey@0"
I20260812 07:04:10.863435 11936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance:
uuid: "8f6cc82b78e44865be6248e20b9e5b8f"
format_stamp: "Formatted at 2026-08-12 07:04:10 on dist-test-slave-69rg"
server_key: "fc630ac47a64e389e0bf7dd603c7022f"
server_key_iv: "c8d07791a9261e771d5cd2748b7f3cf0"
server_key_version: "encryptionkey@0"
I20260812 07:04:10.869966 11936 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.006s
I20260812 07:04:10.874850 11953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:10.876719 11936 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260812 07:04:10.876965 11936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-0/wal
uuid: "8f6cc82b78e44865be6248e20b9e5b8f"
format_stamp: "Formatted at 2026-08-12 07:04:10 on dist-test-slave-69rg"
server_key: "fc630ac47a64e389e0bf7dd603c7022f"
server_key_iv: "c8d07791a9261e771d5cd2748b7f3cf0"
server_key_version: "encryptionkey@0"
I20260812 07:04:10.877275 11936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:10.884331 11936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:10.885403 11936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:10.885763 11936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:10.887141 11936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:10.888993 11936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:10.889286 11936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:10.889559 11936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:10.889675 11936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:10.925063 11936 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.1:46073
I20260812 07:04:10.925118 12085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.1:46073 every 8 connection(s)
I20260812 07:04:10.926522 11936 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
I20260812 07:04:10.931277 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 11936
I20260812 07:04:10.931490 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance
I20260812 07:04:10.932157 6388 external_mini_cluster.cc:1428] Setting key d64920ee504ec9a3ca9557fc29ed2805
I20260812 07:04:10.935274 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:10.929294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:35651 (local address 127.6.61.62:34537)
0812 07:04:10.929746 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:10.929751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:10.930854 (+ 1103us) server_negotiation.cc:407] Connection header received
0812 07:04:10.932730 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:10.932736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:10.932805 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:10.932902 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:10.934416 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:10.934422 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:10.934427 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:04:10.934449 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:10.934506 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:04:10.934523 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:10.934714 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:10.934887 (+ 173us) server_negotiation.cc:299] Negotiation successful
0812 07:04:10.934952 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":266,"threads_started":1}
I20260812 07:04:10.935536 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.2:0
--local_ip_for_outbound_sockets=127.6.61.2
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:34537
--builtin_ntp_servers=127.6.61.20:38557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 07:04:10.936246 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:10.929674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34537 (local address 127.6.61.1:35651)
0812 07:04:10.930632 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:10.930667 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:10.932442 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:10.933159 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:10.933167 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:10.933254 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:10.933992 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:04:10.934014 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:10.934068 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:04:10.934198 (+ 130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:10.934665 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:10.934673 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:10.934679 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 07:04:10.934908 (+ 229us) client_negotiation.cc:240] Negotiation successful
0812 07:04:10.935099 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":375,"threads_started":1}
I20260812 07:04:10.937898 12086 heartbeater.cc:347] Connected to a master server at 127.6.61.62:34537
I20260812 07:04:10.938217 12086 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:10.938831 12086 heartbeater.cc:511] Master 127.6.61.62:34537 requested a full tablet report, sending...
I20260812 07:04:10.941174 11863 ts_manager.cc:194] Registered new tserver with Master: 8f6cc82b78e44865be6248e20b9e5b8f (127.6.61.1:46073)
W20260812 07:04:11.283111 12090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:11.283411 12090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:11.283553 12090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:11.291584 12090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:11.291697 12090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:11.291863 12090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.2
I20260812 07:04:11.301061 12090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:38557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:34537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.2
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:11.302484 12090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:11.304275 12090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:11.318320 12096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:11.318392 12098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:11.319057 12095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:11.319375 12090 server_base.cc:1034] running on GCE node
I20260812 07:04:11.320159 12090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:11.321166 12090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:11.322435 12090 hybrid_clock.cc:648] HybridClock initialized: now 1786518251322356 us; error 61 us; skew 500 ppm
I20260812 07:04:11.322851 12090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 07:04:11.325459 12090 webserver.cc:460] Webserver started at http://127.6.61.2:35293/ using document root <none> and password file <none>
I20260812 07:04:11.326304 12090 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:11.326455 12090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:11.326815 12090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:11.330250 12090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-1/data/instance:
uuid: "a0e5f021cdd54aa69990bd9015fec729"
format_stamp: "Formatted at 2026-08-12 07:04:11 on dist-test-slave-69rg"
server_key: "6cfe34c23d4c1e5e4db32bae4211fc11"
server_key_iv: "4dcf690b1ab7f6748b0eb24580180772"
server_key_version: "encryptionkey@0"
I20260812 07:04:11.331130 12090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance:
uuid: "a0e5f021cdd54aa69990bd9015fec729"
format_stamp: "Formatted at 2026-08-12 07:04:11 on dist-test-slave-69rg"
server_key: "6cfe34c23d4c1e5e4db32bae4211fc11"
server_key_iv: "4dcf690b1ab7f6748b0eb24580180772"
server_key_version: "encryptionkey@0"
I20260812 07:04:11.337086 12090 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260812 07:04:11.342027 12105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:11.343801 12090 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 07:04:11.343986 12090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-1/wal
uuid: "a0e5f021cdd54aa69990bd9015fec729"
format_stamp: "Formatted at 2026-08-12 07:04:11 on dist-test-slave-69rg"
server_key: "6cfe34c23d4c1e5e4db32bae4211fc11"
server_key_iv: "4dcf690b1ab7f6748b0eb24580180772"
server_key_version: "encryptionkey@0"
I20260812 07:04:11.344249 12090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:11.352281 12090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:11.353303 12090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:11.353611 12090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:11.354939 12090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:11.356843 12090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:11.356961 12090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:11.357069 12090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:11.357138 12090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:11.394203 12090 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.2:37917
I20260812 07:04:11.394328 12242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.2:37917 every 8 connection(s)
I20260812 07:04:11.395606 12090 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
I20260812 07:04:11.401944 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 12090
I20260812 07:04:11.402140 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance
I20260812 07:04:11.402643 6388 external_mini_cluster.cc:1428] Setting key 46d41ee81766347467990184683bd63b
I20260812 07:04:11.404255 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:11.398516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:44241 (local address 127.6.61.62:34537)
0812 07:04:11.398736 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:11.398742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:11.400079 (+ 1337us) server_negotiation.cc:407] Connection header received
0812 07:04:11.401794 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:11.401799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:11.401894 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:11.402006 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:11.403527 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:11.403534 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:11.403539 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:04:11.403559 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:11.403605 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:04:11.403621 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:11.403778 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:11.403978 (+ 200us) server_negotiation.cc:299] Negotiation successful
0812 07:04:11.404033 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260812 07:04:11.405341 12246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:11.399005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34537 (local address 127.6.61.2:44241)
0812 07:04:11.399910 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:11.399945 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:11.401521 (+ 1576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:11.402238 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:11.402267 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:11.402368 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:11.403118 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:04:11.403141 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:11.403196 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:04:11.403321 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:11.403756 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:11.403767 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:11.403775 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 07:04:11.404054 (+ 279us) client_negotiation.cc:240] Negotiation successful
0812 07:04:11.404304 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":283,"threads_started":1}
I20260812 07:04:11.405774 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.3:0
--local_ip_for_outbound_sockets=127.6.61.3
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:34537
--builtin_ntp_servers=127.6.61.20:38557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 07:04:11.406726 12244 heartbeater.cc:347] Connected to a master server at 127.6.61.62:34537
I20260812 07:04:11.407022 12244 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:11.407581 12244 heartbeater.cc:511] Master 127.6.61.62:34537 requested a full tablet report, sending...
I20260812 07:04:11.408951 11863 ts_manager.cc:194] Registered new tserver with Master: a0e5f021cdd54aa69990bd9015fec729 (127.6.61.2:37917)
W20260812 07:04:11.749174 12247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:11.749491 12247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:11.749650 12247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:11.757378 12247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:11.757503 12247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:11.757675 12247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.3
I20260812 07:04:11.767376 12247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:38557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:34537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.3
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:11.768833 12247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:11.770737 12247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:11.784462 12252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:11.785235 12253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:11.785320 12255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:11.785648 12247 server_base.cc:1034] running on GCE node
I20260812 07:04:11.786537 12247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:11.787482 12247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:11.788688 12247 hybrid_clock.cc:648] HybridClock initialized: now 1786518251788630 us; error 50 us; skew 500 ppm
I20260812 07:04:11.789003 12247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 07:04:11.791318 12247 webserver.cc:460] Webserver started at http://127.6.61.3:42667/ using document root <none> and password file <none>
I20260812 07:04:11.791987 12247 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:11.792080 12247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:11.792344 12247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:11.795249 12247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-2/data/instance:
uuid: "9ccf55e77bc840e699371f83c6891c69"
format_stamp: "Formatted at 2026-08-12 07:04:11 on dist-test-slave-69rg"
server_key: "0977412c142c5b4b86570a3bbc0c6c87"
server_key_iv: "deba06b80e95c8595b9bd3efab6414f6"
server_key_version: "encryptionkey@0"
I20260812 07:04:11.796010 12247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance:
uuid: "9ccf55e77bc840e699371f83c6891c69"
format_stamp: "Formatted at 2026-08-12 07:04:11 on dist-test-slave-69rg"
server_key: "0977412c142c5b4b86570a3bbc0c6c87"
server_key_iv: "deba06b80e95c8595b9bd3efab6414f6"
server_key_version: "encryptionkey@0"
I20260812 07:04:11.801012 12247 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 07:04:11.804494 12262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:11.805999 12247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 07:04:11.806138 12247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-2/wal
uuid: "9ccf55e77bc840e699371f83c6891c69"
format_stamp: "Formatted at 2026-08-12 07:04:11 on dist-test-slave-69rg"
server_key: "0977412c142c5b4b86570a3bbc0c6c87"
server_key_iv: "deba06b80e95c8595b9bd3efab6414f6"
server_key_version: "encryptionkey@0"
I20260812 07:04:11.806334 12247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:11.811887 12247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:11.812644 12247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:11.812875 12247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:11.813759 12247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:11.815490 12247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:11.815601 12247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:11.815706 12247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:11.815770 12247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:11.853757 12247 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.3:38373
I20260812 07:04:11.853865 12403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.3:38373 every 8 connection(s)
I20260812 07:04:11.855294 12247 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
I20260812 07:04:11.860579 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 12247
I20260812 07:04:11.860780 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance
I20260812 07:04:11.861266 6388 external_mini_cluster.cc:1428] Setting key 235d6b063e067161ac7d2011962646ad
I20260812 07:04:11.864621 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:11.858138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:50661 (local address 127.6.61.62:34537)
0812 07:04:11.858357 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:11.858361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:11.859909 (+ 1548us) server_negotiation.cc:407] Connection header received
0812 07:04:11.862017 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:11.862023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:11.862109 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:11.862252 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:11.863811 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:11.863816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:11.863819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:04:11.863837 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:11.863870 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:04:11.863882 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:11.864014 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:11.864241 (+ 227us) server_negotiation.cc:299] Negotiation successful
0812 07:04:11.864315 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260812 07:04:11.865545 12406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:11.858689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34537 (local address 127.6.61.3:50661)
0812 07:04:11.859736 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:11.859775 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:11.861666 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:11.862508 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:11.862517 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:11.862617 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:11.863476 (+ 859us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:04:11.863479 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:11.863518 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:04:11.863604 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:11.863991 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:11.864002 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:11.864010 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 07:04:11.864342 (+ 332us) client_negotiation.cc:240] Negotiation successful
0812 07:04:11.864606 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":310,"threads_started":1}
I20260812 07:04:11.866925 12404 heartbeater.cc:347] Connected to a master server at 127.6.61.62:34537
I20260812 07:04:11.867175 12404 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:11.867733 12404 heartbeater.cc:511] Master 127.6.61.62:34537 requested a full tablet report, sending...
I20260812 07:04:11.869092 11863 ts_manager.cc:194] Registered new tserver with Master: 9ccf55e77bc840e699371f83c6891c69 (127.6.61.3:38373)
I20260812 07:04:11.877149 6388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 07:04:11.885622 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:11.882491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46870 (local address 127.6.61.62:34537)
0812 07:04:11.882695 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:11.882700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:11.883345 (+ 645us) server_negotiation.cc:407] Connection header received
0812 07:04:11.883565 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:11.883572 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:11.883644 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:11.883761 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:11.884879 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:11.884885 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:11.884891 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:04:11.884912 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:11.884957 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:04:11.884971 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:11.885115 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:11.885200 (+ 85us) server_negotiation.cc:299] Negotiation successful
0812 07:04:11.885271 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 07:04:11.890105 11863 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:04:11.897575 11863 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:04:11.917372 12419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:11.910788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:46073 (local address 127.0.0.1:47068)
0812 07:04:11.912497 (+ 1709us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:11.912552 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:11.912706 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:11.914511 (+ 1805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:11.914520 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:11.914619 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:11.915550 (+ 931us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:04:11.915556 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:11.915595 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:04:11.915701 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:11.916353 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:11.916370 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:11.916381 (+ 11us) client_negotiation.cc:769] Sending connection context
0812 07:04:11.916604 (+ 223us) client_negotiation.cc:240] Negotiation successful
0812 07:04:11.916715 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1426,"thread_start_us":1212,"threads_started":1}
I20260812 07:04:11.918515 12417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:11.910847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47068 (local address 127.6.61.1:46073)
0812 07:04:11.911399 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:11.911409 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:11.912862 (+ 1453us) server_negotiation.cc:407] Connection header received
0812 07:04:11.912944 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:11.912952 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:11.913206 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:11.913382 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:11.915941 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:11.915956 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:11.915976 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:04:11.916003 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:11.916078 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:04:11.916113 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:11.916284 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:11.917858 (+ 1574us) server_negotiation.cc:299] Negotiation successful
0812 07:04:11.918009 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":251,"threads_started":1}
I20260812 07:04:11.924717 12419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:11.912230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:38373 (local address 127.0.0.1:49790)
0812 07:04:11.919050 (+ 6820us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:11.919074 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:11.919179 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:11.921677 (+ 2498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:11.921686 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:11.921752 (+ 66us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:11.922484 (+ 732us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:04:11.922491 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:11.922517 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:04:11.922586 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:11.924154 (+ 1568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:11.924160 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:11.924165 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 07:04:11.924268 (+ 103us) client_negotiation.cc:240] Negotiation successful
0812 07:04:11.924343 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6679,"thread_start_us":2930,"threads_started":1}
I20260812 07:04:11.924074 12011 tablet_service.cc:1467] Processing CreateTablet for tablet 2e55bff6a56245d0b0ef359816bed35a (DEFAULT_TABLE table=test-table [id=884d0871cd984fa4915a0fdfb95be78c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:04:11.925792 12422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:11.913260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49790 (local address 127.6.61.3:38373)
0812 07:04:11.920963 (+ 7703us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:11.920991 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:11.921039 (+ 48us) server_negotiation.cc:407] Connection header received
0812 07:04:11.921133 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:11.921146 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:11.921414 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:11.922185 (+ 771us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:11.923901 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:11.923910 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:11.923925 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:04:11.923943 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:11.924002 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:04:11.924026 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:11.925221 (+ 1195us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:11.925300 (+ 79us) server_negotiation.cc:299] Negotiation successful
0812 07:04:11.925405 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7440,"threads_started":1}
I20260812 07:04:11.926383 12011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e55bff6a56245d0b0ef359816bed35a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:11.928406 12421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:11.910846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:37917 (local address 127.0.0.1:34258)
0812 07:04:11.920694 (+ 9848us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:11.920714 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:11.920809 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:11.924129 (+ 3320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:11.924138 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:11.924164 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:11.926551 (+ 2387us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:04:11.926555 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:11.926573 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:04:11.926620 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:11.927816 (+ 1196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:11.927822 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:11.927828 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 07:04:11.927952 (+ 124us) client_negotiation.cc:240] Negotiation successful
0812 07:04:11.928003 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4494,"threads_started":1}
I20260812 07:04:11.929129 12420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:11.911578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34258 (local address 127.6.61.2:37917)
0812 07:04:11.923351 (+ 11773us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:11.923362 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:11.923398 (+ 36us) server_negotiation.cc:407] Connection header received
0812 07:04:11.923484 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:11.923492 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:11.923701 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:11.926389 (+ 2688us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:11.927567 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:11.927575 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:11.927588 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:04:11.927611 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:11.927658 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:04:11.927675 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:11.928628 (+ 953us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:11.928702 (+ 74us) server_negotiation.cc:299] Negotiation successful
0812 07:04:11.928790 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11549,"thread_start_us":12236,"threads_started":1}
I20260812 07:04:11.933681 12164 tablet_service.cc:1467] Processing CreateTablet for tablet 2e55bff6a56245d0b0ef359816bed35a (DEFAULT_TABLE table=test-table [id=884d0871cd984fa4915a0fdfb95be78c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:04:11.933689 12320 tablet_service.cc:1467] Processing CreateTablet for tablet 2e55bff6a56245d0b0ef359816bed35a (DEFAULT_TABLE table=test-table [id=884d0871cd984fa4915a0fdfb95be78c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:04:11.935256 12164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e55bff6a56245d0b0ef359816bed35a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:11.935253 12320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e55bff6a56245d0b0ef359816bed35a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:11.943050 12423 tablet_bootstrap.cc:492] T 2e55bff6a56245d0b0ef359816bed35a P 8f6cc82b78e44865be6248e20b9e5b8f: Bootstrap starting.
I20260812 07:04:11.944487 12086 heartbeater.cc:503] Master 127.6.61.62:34537 was elected leader, sending a full tablet report...
I20260812 07:04:11.946672 12423 tablet_bootstrap.cc:654] T 2e55bff6a56245d0b0ef359816bed35a P 8f6cc82b78e44865be6248e20b9e5b8f: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:11.948724 12423 log.cc:824] T 2e55bff6a56245d0b0ef359816bed35a P 8f6cc82b78e44865be6248e20b9e5b8f: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:11.952628 12425 tablet_bootstrap.cc:492] T 2e55bff6a56245d0b0ef359816bed35a P 9ccf55e77bc840e699371f83c6891c69: Bootstrap starting.
I20260812 07:04:11.953056 12423 tablet_bootstrap.cc:492] T 2e55bff6a56245d0b0ef359816bed35a P 8f6cc82b78e44865be6248e20b9e5b8f: No bootstrap required, opened a new log
I20260812 07:04:11.953470 12423 ts_tablet_manager.cc:1397] T 2e55bff6a56245d0b0ef359816bed35a P 8f6cc82b78e44865be6248e20b9e5b8f: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260812 07:04:11.954800 12426 tablet_bootstrap.cc:492] T 2e55bff6a56245d0b0ef359816bed35a P a0e5f021cdd54aa69990bd9015fec729: Bootstrap starting.
I20260812 07:04:11.957954 12425 tablet_bootstrap.cc:654] T 2e55bff6a56245d0b0ef359816bed35a P 9ccf55e77bc840e699371f83c6891c69: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:11.959240 12426 tablet_bootstrap.cc:654] T 2e55bff6a56245d0b0ef359816bed35a P a0e5f021cdd54aa69990bd9015fec729: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:11.959439 12425 log.cc:824] T 2e55bff6a56245d0b0ef359816bed35a P 9ccf55e77bc840e699371f83c6891c69: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:11.960561 12426 log.cc:824] T 2e55bff6a56245d0b0ef359816bed35a P a0e5f021cdd54aa69990bd9015fec729: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:11.962513 12423 raft_consensus.cc:348] T 2e55bff6a56245d0b0ef359816bed35a P 8f6cc82b78e44865be6248e20b9e5b8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6cc82b78e44865be6248e20b9e5b8f" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 46073 } } peers { permanent_uuid: "a0e5f021cdd54aa69990bd9015fec729" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 37917 } } peers { permanent_uuid: "9ccf55e77bc840e699371f83c6891c69" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38373 } }
I20260812 07:04:11.963011 12423 raft_consensus.cc:374] T 2e55bff6a56245d0b0ef359816bed35a P 8f6cc82b78e44865be6248e20b9e5b8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:11.963157 12425 tablet_bootstrap.cc:492] T 2e55bff6a56245d0b0ef359816bed35a P 9ccf55e77bc840e699371f83c6891c69: No bootstrap required, opened a new log
I20260812 07:04:11.963169 12423 raft_consensus.cc:729] T 2e55bff6a56245d0b0ef359816bed35a P 8f6cc82b78e44865be6248e20b9e5b8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f6cc82b78e44865be6248e20b9e5b8f, State: Initialized, Role: FOLLOWER
I20260812 07:04:11.963546 12425 ts_tablet_manager.cc:1397] T 2e55bff6a56245d0b0ef359816bed35a P 9ccf55e77bc840e699371f83c6891c69: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260812 07:04:11.963841 12423 consensus_queue.cc:260] T 2e55bff6a56245d0b0ef359816bed35a P 8f6cc82b78e44865be6248e20b9e5b8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6cc82b78e44865be6248e20b9e5b8f" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 46073 } } peers { permanent_uuid: "a0e5f021cdd54aa69990bd9015fec729" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 37917 } } peers { permanent_uuid: "9ccf55e77bc840e699371f83c6891c69" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38373 } }
I20260812 07:04:11.964835 12426 tablet_bootstrap.cc:492] T 2e55bff6a56245d0b0ef359816bed35a P a0e5f021cdd54aa69990bd9015fec729: No bootstrap required, opened a new log
I20260812 07:04:11.965142 12426 ts_tablet_manager.cc:1397] T 2e55bff6a56245d0b0ef359816bed35a P a0e5f021cdd54aa69990bd9015fec729: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260812 07:04:11.965750 12423 ts_tablet_manager.cc:1428] T 2e55bff6a56245d0b0ef359816bed35a P 8f6cc82b78e44865be6248e20b9e5b8f: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260812 07:04:11.973138 12425 raft_consensus.cc:348] T 2e55bff6a56245d0b0ef359816bed35a P 9ccf55e77bc840e699371f83c6891c69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6cc82b78e44865be6248e20b9e5b8f" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 46073 } } peers { permanent_uuid: "a0e5f021cdd54aa69990bd9015fec729" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 37917 } } peers { permanent_uuid: "9ccf55e77bc840e699371f83c6891c69" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38373 } }
I20260812 07:04:11.973614 12425 raft_consensus.cc:374] T 2e55bff6a56245d0b0ef359816bed35a P 9ccf55e77bc840e699371f83c6891c69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:11.973752 12425 raft_consensus.cc:729] T 2e55bff6a56245d0b0ef359816bed35a P 9ccf55e77bc840e699371f83c6891c69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ccf55e77bc840e699371f83c6891c69, State: Initialized, Role: FOLLOWER
I20260812 07:04:11.974001 12426 raft_consensus.cc:348] T 2e55bff6a56245d0b0ef359816bed35a P a0e5f021cdd54aa69990bd9015fec729 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6cc82b78e44865be6248e20b9e5b8f" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 46073 } } peers { permanent_uuid: "a0e5f021cdd54aa69990bd9015fec729" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 37917 } } peers { permanent_uuid: "9ccf55e77bc840e699371f83c6891c69" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38373 } }
I20260812 07:04:11.974452 12426 raft_consensus.cc:374] T 2e55bff6a56245d0b0ef359816bed35a P a0e5f021cdd54aa69990bd9015fec729 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:11.974581 12426 raft_consensus.cc:729] T 2e55bff6a56245d0b0ef359816bed35a P a0e5f021cdd54aa69990bd9015fec729 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e5f021cdd54aa69990bd9015fec729, State: Initialized, Role: FOLLOWER
I20260812 07:04:11.974469 12425 consensus_queue.cc:260] T 2e55bff6a56245d0b0ef359816bed35a P 9ccf55e77bc840e699371f83c6891c69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6cc82b78e44865be6248e20b9e5b8f" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 46073 } } peers { permanent_uuid: "a0e5f021cdd54aa69990bd9015fec729" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 37917 } } peers { permanent_uuid: "9ccf55e77bc840e699371f83c6891c69" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38373 } }
I20260812 07:04:11.975183 12426 consensus_queue.cc:260] T 2e55bff6a56245d0b0ef359816bed35a P a0e5f021cdd54aa69990bd9015fec729 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6cc82b78e44865be6248e20b9e5b8f" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 46073 } } peers { permanent_uuid: "a0e5f021cdd54aa69990bd9015fec729" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 37917 } } peers { permanent_uuid: "9ccf55e77bc840e699371f83c6891c69" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38373 } }
I20260812 07:04:11.975831 12404 heartbeater.cc:503] Master 127.6.61.62:34537 was elected leader, sending a full tablet report...
I20260812 07:04:11.976622 12425 ts_tablet_manager.cc:1428] T 2e55bff6a56245d0b0ef359816bed35a P 9ccf55e77bc840e699371f83c6891c69: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260812 07:04:11.976645 12426 ts_tablet_manager.cc:1428] T 2e55bff6a56245d0b0ef359816bed35a P a0e5f021cdd54aa69990bd9015fec729: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260812 07:04:11.976985 12244 heartbeater.cc:503] Master 127.6.61.62:34537 was elected leader, sending a full tablet report...
I20260812 07:04:12.015689 12432 raft_consensus.cc:482] T 2e55bff6a56245d0b0ef359816bed35a P a0e5f021cdd54aa69990bd9015fec729 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:04:12.016119 12432 raft_consensus.cc:504] T 2e55bff6a56245d0b0ef359816bed35a P a0e5f021cdd54aa69990bd9015fec729 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6cc82b78e44865be6248e20b9e5b8f" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 46073 } } peers { permanent_uuid: "a0e5f021cdd54aa69990bd9015fec729" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 37917 } } peers { permanent_uuid: "9ccf55e77bc840e699371f83c6891c69" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38373 } }
I20260812 07:04:12.018003 12432 leader_election.cc:288] T 2e55bff6a56245d0b0ef359816bed35a P a0e5f021cdd54aa69990bd9015fec729 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f6cc82b78e44865be6248e20b9e5b8f (127.6.61.1:46073), 9ccf55e77bc840e699371f83c6891c69 (127.6.61.3:38373)
I20260812 07:04:12.021864 12422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:12.019052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:55449 (local address 127.6.61.3:38373)
0812 07:04:12.019243 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:12.019262 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:12.019716 (+ 454us) server_negotiation.cc:407] Connection header received
0812 07:04:12.019905 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:12.019909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:12.019958 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:12.020041 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:12.021264 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:12.021268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:12.021271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:04:12.021286 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:12.021319 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:04:12.021332 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:12.021431 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:12.021575 (+ 144us) server_negotiation.cc:299] Negotiation successful
0812 07:04:12.021626 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 07:04:12.022307 12435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:12.018357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:38373 (local address 127.6.61.2:55449)
0812 07:04:12.019591 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:12.019617 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:12.019774 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:12.020116 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:12.020124 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:12.020160 (+ 36us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:12.020989 (+ 829us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:04:12.020995 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:12.021021 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:04:12.021089 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:12.021436 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:12.021444 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:12.021452 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 07:04:12.021640 (+ 188us) client_negotiation.cc:240] Negotiation successful
0812 07:04:12.021711 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1043,"threads_started":1}
I20260812 07:04:12.023867 12345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e55bff6a56245d0b0ef359816bed35a" candidate_uuid: "a0e5f021cdd54aa69990bd9015fec729" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ccf55e77bc840e699371f83c6891c69" is_pre_election: true
I20260812 07:04:12.024416 12417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:12.018588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:58115 (local address 127.6.61.1:46073)
0812 07:04:12.018795 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:12.018803 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:12.022471 (+ 3668us) server_negotiation.cc:407] Connection header received
0812 07:04:12.022692 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:12.022696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:12.022746 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:12.022856 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:12.023770 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:12.023774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:12.023778 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:04:12.023794 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:12.023825 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:04:12.023837 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:12.023933 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:12.024061 (+ 128us) server_negotiation.cc:299] Negotiation successful
0812 07:04:12.024105 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 07:04:12.024492 12345 raft_consensus.cc:2454] T 2e55bff6a56245d0b0ef359816bed35a P 9ccf55e77bc840e699371f83c6891c69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e5f021cdd54aa69990bd9015fec729 in term 0.
I20260812 07:04:12.024616 12433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:12.018827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:46073 (local address 127.6.61.2:58115)
0812 07:04:12.022365 (+ 3538us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:12.022385 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:12.022554 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:12.022878 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:12.022883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:12.022906 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:12.023540 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:04:12.023543 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:12.023558 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:04:12.023604 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:12.023945 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:12.023950 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:12.023954 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 07:04:12.024105 (+ 151us) client_negotiation.cc:240] Negotiation successful
0812 07:04:12.024163 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3398,"thread_start_us":416,"threads_started":1}
I20260812 07:04:12.025275 12109 leader_election.cc:302] T 2e55bff6a56245d0b0ef359816bed35a P a0e5f021cdd54aa69990bd9015fec729 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ccf55e77bc840e699371f83c6891c69, a0e5f021cdd54aa69990bd9015fec729; no voters:
I20260812 07:04:12.025695 12432 raft_consensus.cc:2781] T 2e55bff6a56245d0b0ef359816bed35a P a0e5f021cdd54aa69990bd9015fec729 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:04:12.025874 12432 raft_consensus.cc:482] T 2e55bff6a56245d0b0ef359816bed35a P a0e5f021cdd54aa69990bd9015fec729 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:04:12.025645 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e55bff6a56245d0b0ef359816bed35a" candidate_uuid: "a0e5f021cdd54aa69990bd9015fec729" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f6cc82b78e44865be6248e20b9e5b8f" is_pre_election: true
I20260812 07:04:12.026005 12432 raft_consensus.cc:3037] T 2e55bff6a56245d0b0ef359816bed35a P a0e5f021cdd54aa69990bd9015fec729 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:12.026230 12034 raft_consensus.cc:2454] T 2e55bff6a56245d0b0ef359816bed35a P 8f6cc82b78e44865be6248e20b9e5b8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e5f021cdd54aa69990bd9015fec729 in term 0.
I20260812 07:04:12.028447 12432 raft_consensus.cc:504] T 2e55bff6a56245d0b0ef359816bed35a P a0e5f021cdd54aa69990bd9015fec729 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6cc82b78e44865be6248e20b9e5b8f" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 46073 } } peers { permanent_uuid: "a0e5f021cdd54aa69990bd9015fec729" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 37917 } } peers { permanent_uuid: "9ccf55e77bc840e699371f83c6891c69" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38373 } }
I20260812 07:04:12.029040 12432 leader_election.cc:288] T 2e55bff6a56245d0b0ef359816bed35a P a0e5f021cdd54aa69990bd9015fec729 [CANDIDATE]: Term 1 election: Requested vote from peers 8f6cc82b78e44865be6248e20b9e5b8f (127.6.61.1:46073), 9ccf55e77bc840e699371f83c6891c69 (127.6.61.3:38373)
I20260812 07:04:12.029491 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e55bff6a56245d0b0ef359816bed35a" candidate_uuid: "a0e5f021cdd54aa69990bd9015fec729" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f6cc82b78e44865be6248e20b9e5b8f"
I20260812 07:04:12.029574 12345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e55bff6a56245d0b0ef359816bed35a" candidate_uuid: "a0e5f021cdd54aa69990bd9015fec729" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ccf55e77bc840e699371f83c6891c69"
I20260812 07:04:12.029681 12034 raft_consensus.cc:3037] T 2e55bff6a56245d0b0ef359816bed35a P 8f6cc82b78e44865be6248e20b9e5b8f [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:12.029843 12345 raft_consensus.cc:3037] T 2e55bff6a56245d0b0ef359816bed35a P 9ccf55e77bc840e699371f83c6891c69 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:12.031985 12034 raft_consensus.cc:2454] T 2e55bff6a56245d0b0ef359816bed35a P 8f6cc82b78e44865be6248e20b9e5b8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0e5f021cdd54aa69990bd9015fec729 in term 1.
I20260812 07:04:12.032074 12345 raft_consensus.cc:2454] T 2e55bff6a56245d0b0ef359816bed35a P 9ccf55e77bc840e699371f83c6891c69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0e5f021cdd54aa69990bd9015fec729 in term 1.
I20260812 07:04:12.032485 12108 leader_election.cc:302] T 2e55bff6a56245d0b0ef359816bed35a P a0e5f021cdd54aa69990bd9015fec729 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f6cc82b78e44865be6248e20b9e5b8f, a0e5f021cdd54aa69990bd9015fec729; no voters:
I20260812 07:04:12.032832 12432 raft_consensus.cc:2781] T 2e55bff6a56245d0b0ef359816bed35a P a0e5f021cdd54aa69990bd9015fec729 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:04:12.033236 12432 raft_consensus.cc:686] T 2e55bff6a56245d0b0ef359816bed35a P a0e5f021cdd54aa69990bd9015fec729 [term 1 LEADER]: Becoming Leader. State: Replica: a0e5f021cdd54aa69990bd9015fec729, State: Running, Role: LEADER
I20260812 07:04:12.033761 12432 consensus_queue.cc:237] T 2e55bff6a56245d0b0ef359816bed35a P a0e5f021cdd54aa69990bd9015fec729 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6cc82b78e44865be6248e20b9e5b8f" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 46073 } } peers { permanent_uuid: "a0e5f021cdd54aa69990bd9015fec729" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 37917 } } peers { permanent_uuid: "9ccf55e77bc840e699371f83c6891c69" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38373 } }
I20260812 07:04:12.038761 11861 catalog_manager.cc:5495] T 2e55bff6a56245d0b0ef359816bed35a P a0e5f021cdd54aa69990bd9015fec729 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0e5f021cdd54aa69990bd9015fec729 (127.6.61.2). New cstate: current_term: 1 leader_uuid: "a0e5f021cdd54aa69990bd9015fec729" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f6cc82b78e44865be6248e20b9e5b8f" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 46073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0e5f021cdd54aa69990bd9015fec729" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 37917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ccf55e77bc840e699371f83c6891c69" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38373 } health_report { overall_health: UNKNOWN } } }
I20260812 07:04:12.075832 12420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:12.072726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34268 (local address 127.6.61.2:37917)
0812 07:04:12.072990 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:12.072997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:12.073021 (+ 24us) server_negotiation.cc:407] Connection header received
0812 07:04:12.073277 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:12.073281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:12.073371 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:12.073503 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:12.075128 (+ 1625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:12.075133 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:12.075136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:04:12.075155 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:12.075194 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:04:12.075207 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:12.075357 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:12.075452 (+ 95us) server_negotiation.cc:299] Negotiation successful
0812 07:04:12.075524 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260812 07:04:12.086346 12034 raft_consensus.cc:1261] T 2e55bff6a56245d0b0ef359816bed35a P 8f6cc82b78e44865be6248e20b9e5b8f [term 1 FOLLOWER]: Refusing update from remote peer a0e5f021cdd54aa69990bd9015fec729: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:04:12.086346 12345 raft_consensus.cc:1261] T 2e55bff6a56245d0b0ef359816bed35a P 9ccf55e77bc840e699371f83c6891c69 [term 1 FOLLOWER]: Refusing update from remote peer a0e5f021cdd54aa69990bd9015fec729: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:04:12.087749 12436 consensus_queue.cc:1035] T 2e55bff6a56245d0b0ef359816bed35a P a0e5f021cdd54aa69990bd9015fec729 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f6cc82b78e44865be6248e20b9e5b8f" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 46073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:04:12.088282 12432 consensus_queue.cc:1035] T 2e55bff6a56245d0b0ef359816bed35a P a0e5f021cdd54aa69990bd9015fec729 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ccf55e77bc840e699371f83c6891c69" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:04:12.115163 11861 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:04:12.115485 11861 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:04:12.119295 11861 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 10df99979f384f8b860d425567851c7b: Sending DeleteTablet for 3 replicas of tablet 2e55bff6a56245d0b0ef359816bed35a
I20260812 07:04:12.120510 12164 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e55bff6a56245d0b0ef359816bed35a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:04:12 UTC) from {username='slave'} at 127.0.0.1:34258
I20260812 07:04:12.120493 12011 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e55bff6a56245d0b0ef359816bed35a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:04:12 UTC) from {username='slave'} at 127.0.0.1:47068
I20260812 07:04:12.120867 12320 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e55bff6a56245d0b0ef359816bed35a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:04:12 UTC) from {username='slave'} at 127.0.0.1:49790
I20260812 07:04:12.128086 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 11936
I20260812 07:04:12.136163 12453 tablet_replica.cc:330] T 2e55bff6a56245d0b0ef359816bed35a P 9ccf55e77bc840e699371f83c6891c69: stopping tablet replica
I20260812 07:04:12.136826 12453 raft_consensus.cc:2229] T 2e55bff6a56245d0b0ef359816bed35a P 9ccf55e77bc840e699371f83c6891c69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:04:12.137523 12453 raft_consensus.cc:2258] T 2e55bff6a56245d0b0ef359816bed35a P 9ccf55e77bc840e699371f83c6891c69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:04:12.138309 12417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:12.129093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47082 (local address 127.6.61.1:46073)
0812 07:04:12.129553 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:12.129563 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:12.130339 (+ 776us) server_negotiation.cc:407] Connection header received
0812 07:04:12.130424 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:12.130433 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:12.130578 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:12.130700 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:12.136899 (+ 6199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:12.136910 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:12.136918 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:04:12.136951 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:12.137011 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:04:12.137034 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:12.137180 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:12.137750 (+ 570us) server_negotiation.cc:299] Negotiation successful
0812 07:04:12.137857 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283}
I20260812 07:04:12.139832 12081 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:12.139845 12453 ts_tablet_manager.cc:1905] T 2e55bff6a56245d0b0ef359816bed35a P 9ccf55e77bc840e699371f83c6891c69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:04:12.147202 12453 ts_tablet_manager.cc:1918] T 2e55bff6a56245d0b0ef359816bed35a P 9ccf55e77bc840e699371f83c6891c69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:04:12.147501 12453 log.cc:1196] T 2e55bff6a56245d0b0ef359816bed35a P 9ccf55e77bc840e699371f83c6891c69: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-2/wal/wals/2e55bff6a56245d0b0ef359816bed35a
I20260812 07:04:12.149730 12453 ts_tablet_manager.cc:1939] T 2e55bff6a56245d0b0ef359816bed35a P 9ccf55e77bc840e699371f83c6891c69: Deleting consensus metadata
I20260812 07:04:12.151849 11849 catalog_manager.cc:4841] TS 9ccf55e77bc840e699371f83c6891c69 (127.6.61.3:38373): tablet 2e55bff6a56245d0b0ef359816bed35a (table test-table [id=884d0871cd984fa4915a0fdfb95be78c]) successfully deleted
I20260812 07:04:12.153043 12451 tablet_replica.cc:330] T 2e55bff6a56245d0b0ef359816bed35a P a0e5f021cdd54aa69990bd9015fec729: stopping tablet replica
I20260812 07:04:12.153591 12451 raft_consensus.cc:2229] T 2e55bff6a56245d0b0ef359816bed35a P a0e5f021cdd54aa69990bd9015fec729 [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:04:12.154366 12451 raft_consensus.cc:2258] T 2e55bff6a56245d0b0ef359816bed35a P a0e5f021cdd54aa69990bd9015fec729 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:04:12.156765 12451 ts_tablet_manager.cc:1905] T 2e55bff6a56245d0b0ef359816bed35a P a0e5f021cdd54aa69990bd9015fec729: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:04:12.165483 12451 ts_tablet_manager.cc:1918] T 2e55bff6a56245d0b0ef359816bed35a P a0e5f021cdd54aa69990bd9015fec729: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:04:12.165725 12451 log.cc:1196] T 2e55bff6a56245d0b0ef359816bed35a P a0e5f021cdd54aa69990bd9015fec729: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518208686283-6388-0/minicluster-data/ts-1/wal/wals/2e55bff6a56245d0b0ef359816bed35a
I20260812 07:04:12.166566 12451 ts_tablet_manager.cc:1939] T 2e55bff6a56245d0b0ef359816bed35a P a0e5f021cdd54aa69990bd9015fec729: Deleting consensus metadata
I20260812 07:04:12.168934 11847 catalog_manager.cc:4841] TS a0e5f021cdd54aa69990bd9015fec729 (127.6.61.2:37917): tablet 2e55bff6a56245d0b0ef359816bed35a (table test-table [id=884d0871cd984fa4915a0fdfb95be78c]) successfully deleted
I20260812 07:04:12.251212 12452 tablet_replica.cc:330] T 2e55bff6a56245d0b0ef359816bed35a P 8f6cc82b78e44865be6248e20b9e5b8f: stopping tablet replica
I20260812 07:04:12.251754 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 11936
W20260812 07:04:12.275040 11848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.61.1:46073 (error 108)
I20260812 07:04:12.275615 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 12090
I20260812 07:04:12.277433 12418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:12.276747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:46073 (local address 127.0.0.1:47084)
0812 07:04:12.277018 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:12.277172 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:46073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260812 07:04:12.277699 11848 catalog_manager.cc:4584] TS 8f6cc82b78e44865be6248e20b9e5b8f (127.6.61.1:46073): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2e55bff6a56245d0b0ef359816bed35a: Network error: Client connection negotiation failed: client connection to 127.6.61.1:46073: connect: Connection refused (error 111)
I20260812 07:04:12.278923 12420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:12.276444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34270 (local address 127.6.61.2:37917)
0812 07:04:12.276662 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:12.276670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:12.276795 (+ 125us) server_negotiation.cc:407] Connection header received
0812 07:04:12.277038 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:12.277042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:12.277100 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:12.277211 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:12.278235 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:12.278240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:12.278243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:04:12.278262 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:12.278295 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:04:12.278308 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:12.278409 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:12.278551 (+ 142us) server_negotiation.cc:299] Negotiation successful
0812 07:04:12.278599 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 07:04:12.279733 12237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:12.331631 12418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:12.331054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:46073 (local address 127.0.0.1:47094)
0812 07:04:12.331338 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:12.331450 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:46073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260812 07:04:12.332849 12418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:12.332401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:46073 (local address 127.0.0.1:47106)
0812 07:04:12.332575 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:12.332663 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:46073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260812 07:04:12.393909 12418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:12.393173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:46073 (local address 127.0.0.1:47112)
0812 07:04:12.393537 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:12.393659 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:46073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260812 07:04:12.394868 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 12090
I20260812 07:04:12.395269 12418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:12.394770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:46073 (local address 127.0.0.1:47118)
0812 07:04:12.394960 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:12.395072 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:46073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260812 07:04:12.421506 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 12247
I20260812 07:04:12.424320 12422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:12.422349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49796 (local address 127.6.61.3:38373)
0812 07:04:12.422519 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:12.422524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:12.422611 (+ 87us) server_negotiation.cc:407] Connection header received
0812 07:04:12.422763 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:12.422767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:12.422847 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:12.422936 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:12.423787 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:12.423791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:12.423794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:04:12.423812 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:12.423846 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:04:12.423857 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:12.423961 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:12.424039 (+ 78us) server_negotiation.cc:299] Negotiation successful
0812 07:04:12.424087 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 07:04:12.424958 12392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:12.478423 12418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:12.477935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:46073 (local address 127.0.0.1:47132)
0812 07:04:12.478156 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:12.478275 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:46073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260812 07:04:12.479592 12418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:12.479183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:46073 (local address 127.0.0.1:47134)
0812 07:04:12.479412 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:12.479479 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:46073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260812 07:04:12.513046 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 12247
I20260812 07:04:12.534369 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 11831
I20260812 07:04:12.535602 11899 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:12.641978 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 11831
2026-08-12T07:04:12Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2815 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260812 07:04:12.688810 6388 mini_oidc.cc:152] Starting JWKS server
I20260812 07:04:12.691687 6388 webserver.cc:460] Webserver started at https://127.0.0.1:38937/ using document root <none> and password file <none>
I20260812 07:04:12.691964 6388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 07:04:12.692611 6388 webserver.cc:460] Webserver started at http://127.0.0.1:41523/ using document root <none> and password file <none>
2026-08-12T07:04:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:04:12Z Disabled control of system clock
I20260812 07:04:12.726876 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:41601
--webserver_interface=127.6.61.62
--webserver_port=0
--builtin_ntp_servers=127.6.61.20:46599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:41601
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38937/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518208686283-6388-0/testchainca.cert with env {}
W20260812 07:04:13.070082 12465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:13.070458 12465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:13.070628 12465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:13.077913 12465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:04:13.078104 12465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:13.078197 12465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:13.078289 12465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:04:13.078382 12465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 07:04:13.078476 12465 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 07:04:13.078558 12465 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38937/jwks.json
I20260812 07:04:13.088835 12465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:46599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:41601
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:41601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38937/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518208686283-6388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:13.090272 12465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:13.091578 12465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:13.104101 12475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:13.104202 12478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:13.104161 12473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:13.105010 12465 server_base.cc:1034] running on GCE node
I20260812 07:04:13.105749 12465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:13.107177 12465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:13.108390 12465 hybrid_clock.cc:648] HybridClock initialized: now 1786518253108329 us; error 48 us; skew 500 ppm
I20260812 07:04:13.108853 12465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 07:04:13.111182 12465 webserver.cc:460] Webserver started at http://127.6.61.62:32821/ using document root <none> and password file <none>
I20260812 07:04:13.111914 12465 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:13.112061 12465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:13.112387 12465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:13.115367 12465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "12a195b106db4e61ba7867214489197f"
format_stamp: "Formatted at 2026-08-12 07:04:13 on dist-test-slave-69rg"
server_key: "71b215e9c4566e347af60aa763f71f71"
server_key_iv: "73384acffef9057776419da7e1f6d155"
server_key_version: "encryptionkey@0"
I20260812 07:04:13.116194 12465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "12a195b106db4e61ba7867214489197f"
format_stamp: "Formatted at 2026-08-12 07:04:13 on dist-test-slave-69rg"
server_key: "71b215e9c4566e347af60aa763f71f71"
server_key_iv: "73384acffef9057776419da7e1f6d155"
server_key_version: "encryptionkey@0"
I20260812 07:04:13.121459 12465 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 07:04:13.125550 12486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:13.127063 12465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 07:04:13.127285 12465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "12a195b106db4e61ba7867214489197f"
format_stamp: "Formatted at 2026-08-12 07:04:13 on dist-test-slave-69rg"
server_key: "71b215e9c4566e347af60aa763f71f71"
server_key_iv: "73384acffef9057776419da7e1f6d155"
server_key_version: "encryptionkey@0"
I20260812 07:04:13.127537 12465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:13.150242 12465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:13.151152 12465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:13.151461 12465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:13.170591 12465 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:41601
I20260812 07:04:13.170605 12550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:41601 every 8 connection(s)
I20260812 07:04:13.171952 12465 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:04:13.175666 12551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:13.180193 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 12465
I20260812 07:04:13.180423 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:04:13.181236 6388 external_mini_cluster.cc:1428] Setting key 5b983fc3ee7c441e50dc208d49dd355b
I20260812 07:04:13.187361 12551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12a195b106db4e61ba7867214489197f: Bootstrap starting.
I20260812 07:04:13.191022 12551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12a195b106db4e61ba7867214489197f: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:13.192266 12551 log.cc:824] T 00000000000000000000000000000000 P 12a195b106db4e61ba7867214489197f: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:13.195282 12551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12a195b106db4e61ba7867214489197f: No bootstrap required, opened a new log
I20260812 07:04:13.199796 12555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:13.183201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52630 (local address 127.6.61.62:41601)
0812 07:04:13.184175 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:13.184189 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:13.184232 (+ 43us) server_negotiation.cc:407] Connection header received
0812 07:04:13.186065 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:13.186087 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:13.186520 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:13.186837 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:13.187629 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:13.189030 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:13.191058 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:13.191879 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:13.192033 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:13.197122 (+ 5089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:13.197182 (+ 60us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:13.197198 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:04:13.197794 (+ 596us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:13.197977 (+ 183us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:04:13.198002 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:13.198139 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:13.198365 (+ 226us) server_negotiation.cc:299] Negotiation successful
0812 07:04:13.198732 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":349,"threads_started":1}
I20260812 07:04:13.203953 6388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:04:13.203780 12551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12a195b106db4e61ba7867214489197f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a195b106db4e61ba7867214489197f" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 41601 } }
I20260812 07:04:13.204232 12551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12a195b106db4e61ba7867214489197f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:13.204342 12551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12a195b106db4e61ba7867214489197f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12a195b106db4e61ba7867214489197f, State: Initialized, Role: FOLLOWER
I20260812 07:04:13.204959 12551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12a195b106db4e61ba7867214489197f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a195b106db4e61ba7867214489197f" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 41601 } }
I20260812 07:04:13.205240 12551 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12a195b106db4e61ba7867214489197f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:04:13.205374 12551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12a195b106db4e61ba7867214489197f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:04:13.205544 12551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12a195b106db4e61ba7867214489197f [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:13.208189 12551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12a195b106db4e61ba7867214489197f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a195b106db4e61ba7867214489197f" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 41601 } }
I20260812 07:04:13.208648 12551 leader_election.cc:302] T 00000000000000000000000000000000 P 12a195b106db4e61ba7867214489197f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12a195b106db4e61ba7867214489197f; no voters:
I20260812 07:04:13.209182 12551 leader_election.cc:288] T 00000000000000000000000000000000 P 12a195b106db4e61ba7867214489197f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:04:13.209371 12557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12a195b106db4e61ba7867214489197f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:04:13.210150 12557 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12a195b106db4e61ba7867214489197f [term 1 LEADER]: Becoming Leader. State: Replica: 12a195b106db4e61ba7867214489197f, State: Running, Role: LEADER
I20260812 07:04:13.210706 12557 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12a195b106db4e61ba7867214489197f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a195b106db4e61ba7867214489197f" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 41601 } }
I20260812 07:04:13.211385 12551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12a195b106db4e61ba7867214489197f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:04:13.214056 12560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12a195b106db4e61ba7867214489197f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12a195b106db4e61ba7867214489197f. Latest consensus state: current_term: 1 leader_uuid: "12a195b106db4e61ba7867214489197f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a195b106db4e61ba7867214489197f" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 41601 } } }
I20260812 07:04:13.214594 12560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12a195b106db4e61ba7867214489197f [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:13.214373 12558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12a195b106db4e61ba7867214489197f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12a195b106db4e61ba7867214489197f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a195b106db4e61ba7867214489197f" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 41601 } } }
I20260812 07:04:13.214728 12558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12a195b106db4e61ba7867214489197f [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:13.218935 12566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:04:13.224881 12566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:04:13.237010 12566 catalog_manager.cc:1347] Generated new cluster ID: 3f1c5fb087f74c518b91bca6b594d533
I20260812 07:04:13.237179 12566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:04:13.255313 12566 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:04:13.256778 12566 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:04:13.266824 12566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12a195b106db4e61ba7867214489197f: Generated new TSK 0
I20260812 07:04:13.267807 12566 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 07:04:13.315806 12586 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260812 07:04:13.316093 12586 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 07:04:13.318500 12555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:13.292589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52646 (local address 127.6.61.62:41601)
0812 07:04:13.292801 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:13.292815 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:13.293432 (+ 617us) server_negotiation.cc:407] Connection header received
0812 07:04:13.293848 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:13.293855 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:13.293950 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:13.294111 (+ 161us) server_negotiation.cc:226] Negotiated authn=JWT
0812 07:04:13.294688 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:13.295874 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:13.297487 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:13.298051 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:13.298222 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:13.298815 (+ 593us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 07:04:13.317599 (+ 18784us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38937/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 07:04:13.318158 (+ 559us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52646: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38937/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":64}
W20260812 07:04:13.318742 12555 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52646: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38937/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260812 07:04:13.320227 12585 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:13.292488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:41601 (local address 127.0.0.1:52646)
0812 07:04:13.293271 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:13.293292 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:13.293531 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:13.294182 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:13.294186 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:13.294196 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0812 07:04:13.294532 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:13.294542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:13.296050 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:13.296055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:13.297302 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:13.297316 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:13.298211 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:13.298292 (+ 81us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:13.298449 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:13.298512 (+ 63us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 07:04:13.317992 (+ 19480us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38937/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 07:04:13.318418 (+ 426us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.61.62:41601: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38937/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":432,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.6.61.62:41601: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38937/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260812 07:04:13.322474 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 12465
I20260812 07:04:13.324007 12545 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:13.419075 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 12465
2026-08-12T07:04:13Z chronyd exiting
I20260812 07:04:13.474629 6388 test_util.cc:175] -----------------------------------------------
I20260812 07:04:13.474817 6388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518208686283-6388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (790 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260812 07:04:13.478822 6388 mini_oidc.cc:152] Starting JWKS server
I20260812 07:04:13.480746 6388 webserver.cc:460] Webserver started at https://127.0.0.1:35013/ using document root <none> and password file <none>
I20260812 07:04:13.480988 6388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 07:04:13.481695 6388 webserver.cc:460] Webserver started at http://127.0.0.1:34035/ using document root <none> and password file <none>
2026-08-12T07:04:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:04:13Z Disabled control of system clock
I20260812 07:04:13.517035 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:33767
--webserver_interface=127.6.61.62
--webserver_port=0
--builtin_ntp_servers=127.6.61.20:44749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:33767
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35013/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518208686283-6388-0/testchainca.cert with env {}
W20260812 07:04:13.856036 12594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:13.856349 12594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:13.856439 12594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:13.864830 12594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:04:13.864962 12594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:13.865002 12594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:13.865068 12594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:04:13.865118 12594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 07:04:13.865173 12594 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 07:04:13.865213 12594 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35013/jwks.json
I20260812 07:04:13.875758 12594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:44749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:33767
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:33767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35013/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518208686283-6388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:13.877048 12594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:13.878507 12594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:13.892340 12600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:13.892349 12599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:13.892490 12602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:13.893922 12594 server_base.cc:1034] running on GCE node
I20260812 07:04:13.894805 12594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:13.896286 12594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:13.897521 12594 hybrid_clock.cc:648] HybridClock initialized: now 1786518253897474 us; error 47 us; skew 500 ppm
I20260812 07:04:13.897986 12594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 07:04:13.900415 12594 webserver.cc:460] Webserver started at http://127.6.61.62:40399/ using document root <none> and password file <none>
I20260812 07:04:13.901109 12594 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:13.901203 12594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:13.901487 12594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:13.904493 12594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "b2e9d79e72e342e4958e57013b981f9e"
format_stamp: "Formatted at 2026-08-12 07:04:13 on dist-test-slave-69rg"
server_key: "10d7968177ce47511ee1111a02e08f4b"
server_key_iv: "72d4b26b4cb2ec1ec7976769313544fe"
server_key_version: "encryptionkey@0"
I20260812 07:04:13.905287 12594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "b2e9d79e72e342e4958e57013b981f9e"
format_stamp: "Formatted at 2026-08-12 07:04:13 on dist-test-slave-69rg"
server_key: "10d7968177ce47511ee1111a02e08f4b"
server_key_iv: "72d4b26b4cb2ec1ec7976769313544fe"
server_key_version: "encryptionkey@0"
I20260812 07:04:13.910698 12594 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260812 07:04:13.914220 12612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:13.915668 12594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260812 07:04:13.915864 12594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "b2e9d79e72e342e4958e57013b981f9e"
format_stamp: "Formatted at 2026-08-12 07:04:13 on dist-test-slave-69rg"
server_key: "10d7968177ce47511ee1111a02e08f4b"
server_key_iv: "72d4b26b4cb2ec1ec7976769313544fe"
server_key_version: "encryptionkey@0"
I20260812 07:04:13.916096 12594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:13.938460 12594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:13.939404 12594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:13.939746 12594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:13.959534 12594 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:33767
I20260812 07:04:13.959753 12670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:33767 every 8 connection(s)
I20260812 07:04:13.960958 12594 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:04:13.964953 12671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:13.970403 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 12594
I20260812 07:04:13.970634 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:04:13.971319 6388 external_mini_cluster.cc:1428] Setting key 3afdbcab5de46d7b34cb3b3028caa561
I20260812 07:04:13.976300 12671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2e9d79e72e342e4958e57013b981f9e: Bootstrap starting.
I20260812 07:04:13.980170 12671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2e9d79e72e342e4958e57013b981f9e: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:13.981463 12671 log.cc:824] T 00000000000000000000000000000000 P b2e9d79e72e342e4958e57013b981f9e: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:13.985105 12671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2e9d79e72e342e4958e57013b981f9e: No bootstrap required, opened a new log
I20260812 07:04:13.986011 12675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:13.973434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34428 (local address 127.6.61.62:33767)
0812 07:04:13.974592 (+ 1158us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:13.974602 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:13.974643 (+ 41us) server_negotiation.cc:407] Connection header received
0812 07:04:13.976279 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:13.976306 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:13.976907 (+ 601us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:13.977400 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:13.978107 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:13.979415 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:13.981270 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:13.982172 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:13.982427 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:13.983605 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:13.983612 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:13.983626 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:04:13.984173 (+ 547us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:13.984250 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:04:13.984271 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:13.984361 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:13.984621 (+ 260us) server_negotiation.cc:299] Negotiation successful
0812 07:04:13.984967 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":729,"thread_start_us":496,"threads_started":1}
I20260812 07:04:13.989873 6388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:04:13.993175 12671 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2e9d79e72e342e4958e57013b981f9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e9d79e72e342e4958e57013b981f9e" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 33767 } }
I20260812 07:04:13.993635 12671 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2e9d79e72e342e4958e57013b981f9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:13.993722 12671 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2e9d79e72e342e4958e57013b981f9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2e9d79e72e342e4958e57013b981f9e, State: Initialized, Role: FOLLOWER
I20260812 07:04:13.994287 12671 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2e9d79e72e342e4958e57013b981f9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e9d79e72e342e4958e57013b981f9e" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 33767 } }
I20260812 07:04:13.994498 12671 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2e9d79e72e342e4958e57013b981f9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:04:13.994592 12671 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2e9d79e72e342e4958e57013b981f9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:04:13.994710 12671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2e9d79e72e342e4958e57013b981f9e [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:13.996800 12671 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2e9d79e72e342e4958e57013b981f9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e9d79e72e342e4958e57013b981f9e" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 33767 } }
I20260812 07:04:13.997256 12671 leader_election.cc:302] T 00000000000000000000000000000000 P b2e9d79e72e342e4958e57013b981f9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2e9d79e72e342e4958e57013b981f9e; no voters:
I20260812 07:04:13.997788 12671 leader_election.cc:288] T 00000000000000000000000000000000 P b2e9d79e72e342e4958e57013b981f9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:04:13.997994 12677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2e9d79e72e342e4958e57013b981f9e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:04:13.998852 12677 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2e9d79e72e342e4958e57013b981f9e [term 1 LEADER]: Becoming Leader. State: Replica: b2e9d79e72e342e4958e57013b981f9e, State: Running, Role: LEADER
I20260812 07:04:13.999733 12677 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2e9d79e72e342e4958e57013b981f9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e9d79e72e342e4958e57013b981f9e" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 33767 } }
I20260812 07:04:14.000180 12671 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2e9d79e72e342e4958e57013b981f9e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:04:14.002979 12678 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2e9d79e72e342e4958e57013b981f9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2e9d79e72e342e4958e57013b981f9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e9d79e72e342e4958e57013b981f9e" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 33767 } } }
I20260812 07:04:14.003810 12678 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2e9d79e72e342e4958e57013b981f9e [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:14.003019 12679 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2e9d79e72e342e4958e57013b981f9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2e9d79e72e342e4958e57013b981f9e. Latest consensus state: current_term: 1 leader_uuid: "b2e9d79e72e342e4958e57013b981f9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e9d79e72e342e4958e57013b981f9e" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 33767 } } }
I20260812 07:04:14.006446 12679 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2e9d79e72e342e4958e57013b981f9e [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:14.008469 12687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:04:14.014221 12687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:04:14.022909 12687 catalog_manager.cc:1347] Generated new cluster ID: 6703c7636d33472b8bf78720d5b0de67
I20260812 07:04:14.023048 12687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:04:14.047020 12687 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:04:14.049098 12687 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:04:14.065471 12687 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2e9d79e72e342e4958e57013b981f9e: Generated new TSK 0
I20260812 07:04:14.066674 12687 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 07:04:14.107995 12707 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260812 07:04:14.108140 12707 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 07:04:14.109470 12675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:14.088099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34444 (local address 127.6.61.62:33767)
0812 07:04:14.088328 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:14.088334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:14.088836 (+ 502us) server_negotiation.cc:407] Connection header received
0812 07:04:14.089090 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:14.089098 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:14.089217 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:14.089406 (+ 189us) server_negotiation.cc:226] Negotiated authn=JWT
0812 07:04:14.089915 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:14.091128 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:14.092592 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:14.093327 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:14.093454 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:14.093989 (+ 535us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 07:04:14.108734 (+ 14745us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35013/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 07:04:14.109172 (+ 438us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34444: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35013/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":72}
W20260812 07:04:14.109556 12705 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:14.087923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:33767 (local address 127.0.0.1:34444)
0812 07:04:14.088716 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:14.088734 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:14.088881 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:14.089431 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:14.089435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:14.089445 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0812 07:04:14.089713 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:14.089721 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:14.091320 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:14.091325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:14.092436 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:14.092450 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:14.093454 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:14.093503 (+ 49us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:14.093647 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:14.093688 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 07:04:14.109019 (+ 15331us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35013/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 07:04:14.109228 (+ 209us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.61.62:33767: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35013/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":482,"threads_started":1}
W20260812 07:04:14.109602 12675 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34444: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35013/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260812 07:04:14.111485 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 12594
I20260812 07:04:14.112699 12665 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:14.208348 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 12594
2026-08-12T07:04:14Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (791 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260812 07:04:14.269987 6388 mini_oidc.cc:152] Starting JWKS server
I20260812 07:04:14.272090 6388 webserver.cc:460] Webserver started at https://127.0.0.1:42173/ using document root <none> and password file <none>
I20260812 07:04:14.272367 6388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 07:04:14.273062 6388 webserver.cc:460] Webserver started at http://127.0.0.1:42087/ using document root <none> and password file <none>
2026-08-12T07:04:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:04:14Z Disabled control of system clock
I20260812 07:04:14.308967 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:46499
--webserver_interface=127.6.61.62
--webserver_port=0
--builtin_ntp_servers=127.6.61.20:45137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:46499
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42173/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 07:04:14.646932 12716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:14.647338 12716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:14.647513 12716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:14.655856 12716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:04:14.656034 12716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:14.656126 12716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:14.656252 12716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:04:14.656335 12716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 07:04:14.656430 12716 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 07:04:14.656515 12716 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42173/jwks.json
I20260812 07:04:14.667006 12716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:45137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:46499
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:46499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42173/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:14.668426 12716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:14.670444 12716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:14.682873 12724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:14.682857 12722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:14.682988 12721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:14.687485 12716 server_base.cc:1034] running on GCE node
I20260812 07:04:14.688290 12716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:14.689692 12716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:14.690925 12716 hybrid_clock.cc:648] HybridClock initialized: now 1786518254690900 us; error 44 us; skew 500 ppm
I20260812 07:04:14.691282 12716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 07:04:14.693668 12716 webserver.cc:460] Webserver started at http://127.6.61.62:41441/ using document root <none> and password file <none>
I20260812 07:04:14.694430 12716 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:14.694566 12716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:14.694907 12716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:14.697921 12716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "9bf08a5522394f28bbca12f4961232b1"
format_stamp: "Formatted at 2026-08-12 07:04:14 on dist-test-slave-69rg"
server_key: "d1a30176535a6a9bd95fc5b1d7b02169"
server_key_iv: "70a4e881745be05693b0423013b89bc8"
server_key_version: "encryptionkey@0"
I20260812 07:04:14.698786 12716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "9bf08a5522394f28bbca12f4961232b1"
format_stamp: "Formatted at 2026-08-12 07:04:14 on dist-test-slave-69rg"
server_key: "d1a30176535a6a9bd95fc5b1d7b02169"
server_key_iv: "70a4e881745be05693b0423013b89bc8"
server_key_version: "encryptionkey@0"
I20260812 07:04:14.704273 12716 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 07:04:14.708519 12732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:14.710331 12716 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 07:04:14.710561 12716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "9bf08a5522394f28bbca12f4961232b1"
format_stamp: "Formatted at 2026-08-12 07:04:14 on dist-test-slave-69rg"
server_key: "d1a30176535a6a9bd95fc5b1d7b02169"
server_key_iv: "70a4e881745be05693b0423013b89bc8"
server_key_version: "encryptionkey@0"
I20260812 07:04:14.710826 12716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:14.731742 12716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:14.732654 12716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:14.732963 12716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:14.752864 12716 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:46499
I20260812 07:04:14.752900 12791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:46499 every 8 connection(s)
I20260812 07:04:14.754583 12716 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:04:14.759516 12792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:14.761710 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 12716
I20260812 07:04:14.761981 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:04:14.762835 6388 external_mini_cluster.cc:1428] Setting key fb892b5c797040b1f375ef9bfd9a0b43
I20260812 07:04:14.772697 12792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bf08a5522394f28bbca12f4961232b1: Bootstrap starting.
I20260812 07:04:14.776408 12792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bf08a5522394f28bbca12f4961232b1: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:14.777616 12792 log.cc:824] T 00000000000000000000000000000000 P 9bf08a5522394f28bbca12f4961232b1: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:14.778519 12796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:14.765454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59620 (local address 127.6.61.62:46499)
0812 07:04:14.767005 (+ 1551us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:14.767016 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:14.767059 (+ 43us) server_negotiation.cc:407] Connection header received
0812 07:04:14.768712 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:14.768734 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:14.769213 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:14.769542 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:14.770170 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:14.771477 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:14.773293 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:14.774098 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:14.774234 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:14.775558 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:14.775567 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:14.775580 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:04:14.776203 (+ 623us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:14.776332 (+ 129us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:04:14.776358 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:14.776481 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:14.776804 (+ 323us) server_negotiation.cc:299] Negotiation successful
0812 07:04:14.777197 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1120,"thread_start_us":827,"threads_started":1}
I20260812 07:04:14.780738 12792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bf08a5522394f28bbca12f4961232b1: No bootstrap required, opened a new log
I20260812 07:04:14.782794 6388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:04:14.789369 12792 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bf08a5522394f28bbca12f4961232b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf08a5522394f28bbca12f4961232b1" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 46499 } }
I20260812 07:04:14.789716 12792 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bf08a5522394f28bbca12f4961232b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:14.789791 12792 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bf08a5522394f28bbca12f4961232b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bf08a5522394f28bbca12f4961232b1, State: Initialized, Role: FOLLOWER
I20260812 07:04:14.790410 12792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bf08a5522394f28bbca12f4961232b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf08a5522394f28bbca12f4961232b1" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 46499 } }
I20260812 07:04:14.790609 12792 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bf08a5522394f28bbca12f4961232b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:04:14.790709 12792 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bf08a5522394f28bbca12f4961232b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:04:14.790827 12792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bf08a5522394f28bbca12f4961232b1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:14.793041 12792 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bf08a5522394f28bbca12f4961232b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf08a5522394f28bbca12f4961232b1" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 46499 } }
I20260812 07:04:14.793509 12792 leader_election.cc:302] T 00000000000000000000000000000000 P 9bf08a5522394f28bbca12f4961232b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bf08a5522394f28bbca12f4961232b1; no voters:
I20260812 07:04:14.794158 12792 leader_election.cc:288] T 00000000000000000000000000000000 P 9bf08a5522394f28bbca12f4961232b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:04:14.794312 12799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bf08a5522394f28bbca12f4961232b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:04:14.795113 12799 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bf08a5522394f28bbca12f4961232b1 [term 1 LEADER]: Becoming Leader. State: Replica: 9bf08a5522394f28bbca12f4961232b1, State: Running, Role: LEADER
I20260812 07:04:14.795715 12799 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bf08a5522394f28bbca12f4961232b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf08a5522394f28bbca12f4961232b1" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 46499 } }
I20260812 07:04:14.796430 12792 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bf08a5522394f28bbca12f4961232b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:04:14.799665 12801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bf08a5522394f28bbca12f4961232b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bf08a5522394f28bbca12f4961232b1. Latest consensus state: current_term: 1 leader_uuid: "9bf08a5522394f28bbca12f4961232b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf08a5522394f28bbca12f4961232b1" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 46499 } } }
I20260812 07:04:14.800205 12801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bf08a5522394f28bbca12f4961232b1 [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:14.799728 12800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bf08a5522394f28bbca12f4961232b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bf08a5522394f28bbca12f4961232b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf08a5522394f28bbca12f4961232b1" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 46499 } } }
I20260812 07:04:14.802388 12800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bf08a5522394f28bbca12f4961232b1 [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:14.802773 12807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:04:14.808115 12807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:04:14.816792 12807 catalog_manager.cc:1347] Generated new cluster ID: 2e808e0a04c440cbac0d0ed2cb91b4da
I20260812 07:04:14.816905 12807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:04:14.840708 12807 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:04:14.842163 12807 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:04:14.855062 12807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bf08a5522394f28bbca12f4961232b1: Generated new TSK 0
I20260812 07:04:14.856065 12807 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:04:14.882660 12796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:14.874451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59622 (local address 127.6.61.62:46499)
0812 07:04:14.874680 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:14.874688 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:14.875067 (+ 379us) server_negotiation.cc:407] Connection header received
0812 07:04:14.875314 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:14.875319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:14.875405 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:14.875540 (+ 135us) server_negotiation.cc:226] Negotiated authn=JWT
0812 07:04:14.876099 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:14.876899 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:14.878405 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:14.878870 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:14.879018 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:14.879542 (+ 524us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 07:04:14.881986 (+ 2444us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42173/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 07:04:14.882331 (+ 345us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59622: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42173/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":87}
W20260812 07:04:14.882795 12796 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59622: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42173/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260812 07:04:14.884807 12826 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:14.874238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:46499 (local address 127.0.0.1:59622)
0812 07:04:14.874955 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:14.874975 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:14.875139 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:14.875597 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:14.875603 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:14.875617 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0812 07:04:14.875941 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:14.875954 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:14.877061 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:14.877067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:14.878245 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:14.878258 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:14.879005 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:14.879089 (+ 84us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:14.879231 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:14.879267 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 07:04:14.882232 (+ 2965us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42173/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 07:04:14.884326 (+ 2094us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.61.62:46499: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42173/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":1730,"thread_start_us":361,"threads_started":1}
I20260812 07:04:14.886199 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 12716
I20260812 07:04:14.887455 12787 generic_service.cc:196] Checking for leaks (request via RPC)
E20260812 07:04:14.894855 12828 webserver.cc:533] Webserver: sslize failed:
E20260812 07:04:14.894958 12828 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 07:04:14.980214 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 12716
2026-08-12T07:04:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (769 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-12T07:04:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:04:15Z Disabled control of system clock
I20260812 07:04:15.073298 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:43379
--webserver_interface=127.6.61.62
--webserver_port=0
--builtin_ntp_servers=127.6.61.20:34993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:43379
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518208686283-6388-0/testchainca.cert with env {}
W20260812 07:04:15.413126 12836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:15.413443 12836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:15.413605 12836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:15.421005 12836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:04:15.421113 12836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:15.421208 12836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:15.421289 12836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:04:15.421403 12836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 07:04:15.421517 12836 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 07:04:15.421576 12836 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260812 07:04:15.431773 12836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:34993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:43379
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:43379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518208686283-6388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:15.433105 12836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:15.434413 12836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:15.446655 12841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:15.446703 12842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:15.446976 12844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:15.447885 12836 server_base.cc:1034] running on GCE node
I20260812 07:04:15.448629 12836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:15.450138 12836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:15.451382 12836 hybrid_clock.cc:648] HybridClock initialized: now 1786518255451343 us; error 39 us; skew 500 ppm
I20260812 07:04:15.451746 12836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 07:04:15.454236 12836 webserver.cc:460] Webserver started at http://127.6.61.62:43621/ using document root <none> and password file <none>
I20260812 07:04:15.454967 12836 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:15.455109 12836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:15.455466 12836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:15.458406 12836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "07180a5408454f8a8cd46ea272e50f43"
format_stamp: "Formatted at 2026-08-12 07:04:15 on dist-test-slave-69rg"
server_key: "9c973d967167615225a9d2a996f609c4"
server_key_iv: "66873a3fdd9f63850e5622caa6479e15"
server_key_version: "encryptionkey@0"
I20260812 07:04:15.459170 12836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "07180a5408454f8a8cd46ea272e50f43"
format_stamp: "Formatted at 2026-08-12 07:04:15 on dist-test-slave-69rg"
server_key: "9c973d967167615225a9d2a996f609c4"
server_key_iv: "66873a3fdd9f63850e5622caa6479e15"
server_key_version: "encryptionkey@0"
I20260812 07:04:15.464483 12836 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260812 07:04:15.467880 12853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:15.469218 12836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 07:04:15.469413 12836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "07180a5408454f8a8cd46ea272e50f43"
format_stamp: "Formatted at 2026-08-12 07:04:15 on dist-test-slave-69rg"
server_key: "9c973d967167615225a9d2a996f609c4"
server_key_iv: "66873a3fdd9f63850e5622caa6479e15"
server_key_version: "encryptionkey@0"
I20260812 07:04:15.469609 12836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:15.484593 12836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:15.485455 12836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:15.485757 12836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:15.504896 12836 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:43379
I20260812 07:04:15.504904 12914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:43379 every 8 connection(s)
I20260812 07:04:15.506367 12836 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:04:15.507560 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 12836
I20260812 07:04:15.507771 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:04:15.508486 6388 external_mini_cluster.cc:1428] Setting key b6bd17bc5b4d4b780f83f883bcdc23ee
I20260812 07:04:15.510934 12915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:15.524492 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:15.510466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48590 (local address 127.6.61.62:43379)
0812 07:04:15.511557 (+ 1091us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:15.511572 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:15.511623 (+ 51us) server_negotiation.cc:407] Connection header received
0812 07:04:15.513957 (+ 2334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:15.513987 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:15.514616 (+ 629us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:15.515060 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:15.515614 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:15.517725 (+ 2111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:15.519107 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:15.520144 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:15.520344 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:15.521515 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:15.521525 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:15.521544 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:04:15.522347 (+ 803us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:15.522459 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:04:15.522487 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:15.522622 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:15.522823 (+ 201us) server_negotiation.cc:299] Negotiation successful
0812 07:04:15.523196 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":344,"threads_started":1}
I20260812 07:04:15.525569 12915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07180a5408454f8a8cd46ea272e50f43: Bootstrap starting.
I20260812 07:04:15.528445 6388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:04:15.529336 12915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07180a5408454f8a8cd46ea272e50f43: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:15.530632 12915 log.cc:824] T 00000000000000000000000000000000 P 07180a5408454f8a8cd46ea272e50f43: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:15.533731 12915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07180a5408454f8a8cd46ea272e50f43: No bootstrap required, opened a new log
I20260812 07:04:15.541360 12915 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07180a5408454f8a8cd46ea272e50f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07180a5408454f8a8cd46ea272e50f43" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 43379 } }
I20260812 07:04:15.541721 12915 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07180a5408454f8a8cd46ea272e50f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:15.541805 12915 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07180a5408454f8a8cd46ea272e50f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07180a5408454f8a8cd46ea272e50f43, State: Initialized, Role: FOLLOWER
I20260812 07:04:15.542421 12915 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07180a5408454f8a8cd46ea272e50f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07180a5408454f8a8cd46ea272e50f43" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 43379 } }
I20260812 07:04:15.542635 12915 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07180a5408454f8a8cd46ea272e50f43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:04:15.542733 12915 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07180a5408454f8a8cd46ea272e50f43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:04:15.542876 12915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07180a5408454f8a8cd46ea272e50f43 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:15.545166 12915 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07180a5408454f8a8cd46ea272e50f43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07180a5408454f8a8cd46ea272e50f43" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 43379 } }
I20260812 07:04:15.545624 12915 leader_election.cc:302] T 00000000000000000000000000000000 P 07180a5408454f8a8cd46ea272e50f43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07180a5408454f8a8cd46ea272e50f43; no voters:
I20260812 07:04:15.546229 12915 leader_election.cc:288] T 00000000000000000000000000000000 P 07180a5408454f8a8cd46ea272e50f43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:04:15.546398 12920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07180a5408454f8a8cd46ea272e50f43 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:04:15.547118 12920 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07180a5408454f8a8cd46ea272e50f43 [term 1 LEADER]: Becoming Leader. State: Replica: 07180a5408454f8a8cd46ea272e50f43, State: Running, Role: LEADER
I20260812 07:04:15.547659 12920 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07180a5408454f8a8cd46ea272e50f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07180a5408454f8a8cd46ea272e50f43" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 43379 } }
I20260812 07:04:15.548385 12915 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07180a5408454f8a8cd46ea272e50f43 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:04:15.551537 12923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07180a5408454f8a8cd46ea272e50f43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07180a5408454f8a8cd46ea272e50f43. Latest consensus state: current_term: 1 leader_uuid: "07180a5408454f8a8cd46ea272e50f43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07180a5408454f8a8cd46ea272e50f43" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 43379 } } }
I20260812 07:04:15.552217 12923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07180a5408454f8a8cd46ea272e50f43 [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:15.551265 12921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07180a5408454f8a8cd46ea272e50f43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07180a5408454f8a8cd46ea272e50f43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07180a5408454f8a8cd46ea272e50f43" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 43379 } } }
I20260812 07:04:15.552877 12921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07180a5408454f8a8cd46ea272e50f43 [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:15.554543 12930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:04:15.559582 12930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:04:15.568990 12930 catalog_manager.cc:1347] Generated new cluster ID: 815a8358ea064fe682f22c5bcbec079a
I20260812 07:04:15.569120 12930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:04:15.577937 12930 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:04:15.579213 12930 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:04:15.586395 12930 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07180a5408454f8a8cd46ea272e50f43: Generated new TSK 0
I20260812 07:04:15.587208 12930 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:04:15.626864 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:15.605706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48604 (local address 127.6.61.62:43379)
0812 07:04:15.606007 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:15.606014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:15.606429 (+ 415us) server_negotiation.cc:407] Connection header received
0812 07:04:15.606700 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:15.606704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:15.606781 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:15.606917 (+ 136us) server_negotiation.cc:226] Negotiated authn=JWT
0812 07:04:15.607568 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:15.608286 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:15.610402 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:15.610861 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:15.610955 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:15.611556 (+ 601us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 07:04:15.626279 (+ 14723us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 07:04:15.626624 (+ 345us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48604: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":85}
W20260812 07:04:15.627038 12918 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48604: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260812 07:04:15.627408 12946 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:15.605622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43379 (local address 127.0.0.1:48604)
0812 07:04:15.606309 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:15.606332 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:15.606500 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:15.606976 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:15.606982 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:15.606996 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0812 07:04:15.607396 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:15.607412 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:15.608459 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:15.608465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:15.610254 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:15.610271 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:15.611004 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:15.611061 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:15.611248 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:15.611297 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 07:04:15.626602 (+ 15305us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 07:04:15.626987 (+ 385us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.61.62:43379: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":359,"threads_started":1}
I20260812 07:04:15.628891 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 12836
I20260812 07:04:15.630196 12909 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:15.722951 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 12836
2026-08-12T07:04:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (722 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (433 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (429 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:04:16 dist-test-slave-69rg krb5kdc[12965](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:04:16 dist-test-slave-69rg krb5kdc[12965](info): set up 2 sockets
Aug 12 07:04:16 dist-test-slave-69rg krb5kdc[12965](info): commencing operation
krb5kdc: starting...
W20260812 07:04:17.806581 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.179s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:04:17 dist-test-slave-69rg krb5kdc[12965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:04:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:04:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
I20260812 07:04:18.062924 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:33245
--webserver_interface=127.6.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:38837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:33245
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:18.399955 12981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:18.400236 12981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:18.400411 12981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:18.407683 12981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:04:18.407810 12981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:18.407905 12981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:18.407998 12981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:04:18.408087 12981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:04:18.418288 12981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:38837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:33245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:33245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:18.419682 12981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:18.421046 12981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:18.433105 12986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:18.433473 12987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:18.433624 12990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:18.434494 12981 server_base.cc:1034] running on GCE node
I20260812 07:04:18.435263 12981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:18.436713 12981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:18.437968 12981 hybrid_clock.cc:648] HybridClock initialized: now 1786518258437909 us; error 58 us; skew 500 ppm
I20260812 07:04:18.438409 12981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:04:18 dist-test-slave-69rg krb5kdc[12965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518258, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:18.444703 12981 init.cc:377] Logged in from keytab as kudu/127.6.61.62@KRBTEST.COM (short username kudu)
I20260812 07:04:18.446408 12981 webserver.cc:460] Webserver started at http://127.6.61.62:46447/ using document root <none> and password file <none>
I20260812 07:04:18.447149 12981 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:18.447347 12981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:18.447633 12981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:18.450702 12981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "12f1e5a3ef624052851759976ecabd90"
format_stamp: "Formatted at 2026-08-12 07:04:18 on dist-test-slave-69rg"
server_key: "c0eba2f94e57debd70acee90486b71ac"
server_key_iv: "e293cfdbd05c3ef843b7424344b584ef"
server_key_version: "encryptionkey@0"
I20260812 07:04:18.451491 12981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "12f1e5a3ef624052851759976ecabd90"
format_stamp: "Formatted at 2026-08-12 07:04:18 on dist-test-slave-69rg"
server_key: "c0eba2f94e57debd70acee90486b71ac"
server_key_iv: "e293cfdbd05c3ef843b7424344b584ef"
server_key_version: "encryptionkey@0"
I20260812 07:04:18.457525 12981 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 07:04:18.461203 13001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:18.462724 12981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 07:04:18.462917 12981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "12f1e5a3ef624052851759976ecabd90"
format_stamp: "Formatted at 2026-08-12 07:04:18 on dist-test-slave-69rg"
server_key: "c0eba2f94e57debd70acee90486b71ac"
server_key_iv: "e293cfdbd05c3ef843b7424344b584ef"
server_key_version: "encryptionkey@0"
I20260812 07:04:18.463121 12981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:18.483898 12981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:18.488339 12981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:18.488624 12981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:18.507222 12981 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:33245
I20260812 07:04:18.507314 13057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:33245 every 8 connection(s)
I20260812 07:04:18.508617 12981 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:04:18.512427 13058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:18.518891 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 12981
I20260812 07:04:18.519137 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:04:18.519817 6388 external_mini_cluster.cc:1428] Setting key eac188d3647df4975a86c4ba62415b86
I20260812 07:04:18.523111 13058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12f1e5a3ef624052851759976ecabd90: Bootstrap starting.
I20260812 07:04:18.526569 13058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12f1e5a3ef624052851759976ecabd90: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:18.527774 13058 log.cc:824] T 00000000000000000000000000000000 P 12f1e5a3ef624052851759976ecabd90: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:18.531003 13058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12f1e5a3ef624052851759976ecabd90: No bootstrap required, opened a new log
Aug 12 07:04:18 dist-test-slave-69rg krb5kdc[12965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:04:18.540306 13058 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12f1e5a3ef624052851759976ecabd90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f1e5a3ef624052851759976ecabd90" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 33245 } }
I20260812 07:04:18.540684 13058 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12f1e5a3ef624052851759976ecabd90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:18.540813 13058 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12f1e5a3ef624052851759976ecabd90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12f1e5a3ef624052851759976ecabd90, State: Initialized, Role: FOLLOWER
I20260812 07:04:18.541499 13058 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12f1e5a3ef624052851759976ecabd90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f1e5a3ef624052851759976ecabd90" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 33245 } }
I20260812 07:04:18.541774 13058 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12f1e5a3ef624052851759976ecabd90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:04:18.541970 13058 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12f1e5a3ef624052851759976ecabd90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:04:18.542177 13058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12f1e5a3ef624052851759976ecabd90 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:18.544637 13061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:18.521953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46486 (local address 127.6.61.62:33245)
0812 07:04:18.522777 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:18.522788 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:18.522829 (+ 41us) server_negotiation.cc:407] Connection header received
0812 07:04:18.524632 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:18.524655 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:18.525027 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:18.525332 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:18.527242 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:18.529095 (+ 1853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:18.530901 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:18.531664 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:18.531819 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:18.536803 (+ 4984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:18.536811 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:18.536828 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:18.536867 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:18.540136 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:18.540948 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:18.540959 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:18.540968 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:18.541156 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:18.541583 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:18.541588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:18.541591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:18.542331 (+ 740us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:18.542644 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:18.543180 (+ 536us) server_negotiation.cc:299] Negotiation successful
0812 07:04:18.543648 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":391,"threads_started":1}
I20260812 07:04:18.544616 13058 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12f1e5a3ef624052851759976ecabd90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f1e5a3ef624052851759976ecabd90" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 33245 } }
I20260812 07:04:18.545153 13058 leader_election.cc:302] T 00000000000000000000000000000000 P 12f1e5a3ef624052851759976ecabd90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12f1e5a3ef624052851759976ecabd90; no voters:
I20260812 07:04:18.545806 13058 leader_election.cc:288] T 00000000000000000000000000000000 P 12f1e5a3ef624052851759976ecabd90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:04:18.545984 13063 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12f1e5a3ef624052851759976ecabd90 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:04:18.547446 13063 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12f1e5a3ef624052851759976ecabd90 [term 1 LEADER]: Becoming Leader. State: Replica: 12f1e5a3ef624052851759976ecabd90, State: Running, Role: LEADER
I20260812 07:04:18.548085 13063 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12f1e5a3ef624052851759976ecabd90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f1e5a3ef624052851759976ecabd90" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 33245 } }
I20260812 07:04:18.548269 13058 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12f1e5a3ef624052851759976ecabd90 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:04:18.552572 13065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12f1e5a3ef624052851759976ecabd90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12f1e5a3ef624052851759976ecabd90. Latest consensus state: current_term: 1 leader_uuid: "12f1e5a3ef624052851759976ecabd90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f1e5a3ef624052851759976ecabd90" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 33245 } } }
I20260812 07:04:18.552627 13064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12f1e5a3ef624052851759976ecabd90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12f1e5a3ef624052851759976ecabd90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f1e5a3ef624052851759976ecabd90" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 33245 } } }
I20260812 07:04:18.553035 13065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12f1e5a3ef624052851759976ecabd90 [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:18.553035 13064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12f1e5a3ef624052851759976ecabd90 [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:18.554530 13075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:04:18.559417 13075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:04:18.569205 13075 catalog_manager.cc:1347] Generated new cluster ID: 376fa2e9e19647e981be8ef133cd4959
I20260812 07:04:18.569352 13075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:04:18.578904 13075 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:04:18.580466 13075 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:04:18.586124 13075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12f1e5a3ef624052851759976ecabd90: Generated new TSK 0
I20260812 07:04:18.586907 13075 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
I20260812 07:04:18.650528 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.1:0
--local_ip_for_outbound_sockets=127.6.61.1
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:33245
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:38837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:18.994885 13093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:18.995164 13093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:18.995337 13093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:19.002569 13093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:19.002731 13093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:19.002938 13093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.1
I20260812 07:04:19.012863 13093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:38837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:33245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.1
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:19.014246 13093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:19.015616 13093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:19.028748 13099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:19.028860 13100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:19.029444 13102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:19.029994 13093 server_base.cc:1034] running on GCE node
I20260812 07:04:19.030601 13093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:19.031472 13093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:19.032694 13093 hybrid_clock.cc:648] HybridClock initialized: now 1786518259032654 us; error 26 us; skew 500 ppm
I20260812 07:04:19.032968 13093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:04:19 dist-test-slave-69rg krb5kdc[12965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518259, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:19.036343 13093 init.cc:377] Logged in from keytab as kudu/127.6.61.1@KRBTEST.COM (short username kudu)
I20260812 07:04:19.037855 13093 webserver.cc:460] Webserver started at http://127.6.61.1:46309/ using document root <none> and password file <none>
I20260812 07:04:19.038568 13093 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:19.038707 13093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:19.039039 13093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:19.042029 13093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-0/data/instance:
uuid: "cde225efea734891adece131c92e4e57"
format_stamp: "Formatted at 2026-08-12 07:04:19 on dist-test-slave-69rg"
server_key: "3c053b5c7f8354dce41e6c8a49f6c722"
server_key_iv: "cafa6a42521f3d43d2c804067efb517b"
server_key_version: "encryptionkey@0"
I20260812 07:04:19.042877 13093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance:
uuid: "cde225efea734891adece131c92e4e57"
format_stamp: "Formatted at 2026-08-12 07:04:19 on dist-test-slave-69rg"
server_key: "3c053b5c7f8354dce41e6c8a49f6c722"
server_key_iv: "cafa6a42521f3d43d2c804067efb517b"
server_key_version: "encryptionkey@0"
I20260812 07:04:19.048743 13093 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 07:04:19.052306 13111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:19.053644 13093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 07:04:19.053905 13093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-0/wal
uuid: "cde225efea734891adece131c92e4e57"
format_stamp: "Formatted at 2026-08-12 07:04:19 on dist-test-slave-69rg"
server_key: "3c053b5c7f8354dce41e6c8a49f6c722"
server_key_iv: "cafa6a42521f3d43d2c804067efb517b"
server_key_version: "encryptionkey@0"
I20260812 07:04:19.054133 13093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:19.069731 13093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:19.074251 13093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:19.074558 13093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:19.075553 13093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:19.076934 13093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:19.077041 13093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:19.077198 13093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:19.077281 13093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:19.111964 13093 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.1:36155
I20260812 07:04:19.112008 13242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.1:36155 every 8 connection(s)
I20260812 07:04:19.113348 13093 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
I20260812 07:04:19.117650 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 13093
I20260812 07:04:19.117934 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance
I20260812 07:04:19.118616 6388 external_mini_cluster.cc:1428] Setting key 162f117655a97ef6ce3446a063dced08
Aug 12 07:04:19 dist-test-slave-69rg krb5kdc[12965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518259, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:04:19.135622 13245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:19.115967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:51881 (local address 127.6.61.62:33245)
0812 07:04:19.116503 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:19.116509 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:19.117419 (+ 910us) server_negotiation.cc:407] Connection header received
0812 07:04:19.119521 (+ 2102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:19.119527 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:19.119623 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:19.119752 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:19.121565 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:19.122823 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:19.124382 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:19.125067 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:19.125218 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:19.129578 (+ 4360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:19.129587 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:19.129596 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:19.129641 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:19.132292 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:19.133220 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:19.133224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:19.133227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:19.133354 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:19.133909 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:19.133912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:19.133914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:19.134172 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:19.134301 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:19.135156 (+ 855us) server_negotiation.cc:299] Negotiation successful
0812 07:04:19.135364 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":238,"threads_started":1}
I20260812 07:04:19.137153 13246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:19.116458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:33245 (local address 127.6.61.1:51881)
0812 07:04:19.117241 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:19.117296 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:19.119178 (+ 1882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:19.120030 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:19.120041 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:19.120835 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:19.121383 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:19.121400 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:19.122986 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:19.122990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:19.124226 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:19.124239 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:19.125289 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:19.125294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:19.125491 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:19.126241 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:19.126286 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:19.129337 (+ 3051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:19.132431 (+ 3094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:19.132443 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:19.132463 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:19.133103 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:19.133468 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:19.133475 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:19.133481 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:19.133768 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:19.134308 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:19.134318 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:19.134758 (+ 440us) client_negotiation.cc:769] Sending connection context
0812 07:04:19.135064 (+ 306us) client_negotiation.cc:240] Negotiation successful
0812 07:04:19.135648 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":316,"threads_started":1}
I20260812 07:04:19.139163 13243 heartbeater.cc:347] Connected to a master server at 127.6.61.62:33245
I20260812 07:04:19.139510 13243 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:19.140287 13243 heartbeater.cc:511] Master 127.6.61.62:33245 requested a full tablet report, sending...
I20260812 07:04:19.142755 13018 ts_manager.cc:194] Registered new tserver with Master: cde225efea734891adece131c92e4e57 (127.6.61.1:36155)
I20260812 07:04:19.145058 13018 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.1@KRBTEST.COM'} at 127.6.61.1:51881
WARNING: no policy specified for kudu/127.6.61.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
I20260812 07:04:19.177196 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.2:0
--local_ip_for_outbound_sockets=127.6.61.2
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:33245
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:38837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:19.536085 13250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:19.536423 13250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:19.536592 13250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:19.544108 13250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:19.544272 13250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:19.544445 13250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.2
I20260812 07:04:19.555023 13250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:38837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:33245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.2
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:19.556383 13250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:19.557714 13250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:19.571684 13255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:19.571593 13259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:19.571650 13257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:19.572722 13250 server_base.cc:1034] running on GCE node
I20260812 07:04:19.573352 13250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:19.574380 13250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:19.575620 13250 hybrid_clock.cc:648] HybridClock initialized: now 1786518259575562 us; error 49 us; skew 500 ppm
I20260812 07:04:19.575954 13250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:04:19 dist-test-slave-69rg krb5kdc[12965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518259, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:19.579281 13250 init.cc:377] Logged in from keytab as kudu/127.6.61.2@KRBTEST.COM (short username kudu)
I20260812 07:04:19.580762 13250 webserver.cc:460] Webserver started at http://127.6.61.2:34915/ using document root <none> and password file <none>
I20260812 07:04:19.581403 13250 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:19.581511 13250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:19.581785 13250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:19.584790 13250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-1/data/instance:
uuid: "ec8cba52ec024bb3a945561a11ae03e0"
format_stamp: "Formatted at 2026-08-12 07:04:19 on dist-test-slave-69rg"
server_key: "9477f254c375789fd7c234621231e3e3"
server_key_iv: "963d7a7cb3523e2cad52d6251a50edb1"
server_key_version: "encryptionkey@0"
I20260812 07:04:19.585526 13250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance:
uuid: "ec8cba52ec024bb3a945561a11ae03e0"
format_stamp: "Formatted at 2026-08-12 07:04:19 on dist-test-slave-69rg"
server_key: "9477f254c375789fd7c234621231e3e3"
server_key_iv: "963d7a7cb3523e2cad52d6251a50edb1"
server_key_version: "encryptionkey@0"
I20260812 07:04:19.590632 13250 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260812 07:04:19.593956 13266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:19.595291 13250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 07:04:19.595425 13250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-1/wal
uuid: "ec8cba52ec024bb3a945561a11ae03e0"
format_stamp: "Formatted at 2026-08-12 07:04:19 on dist-test-slave-69rg"
server_key: "9477f254c375789fd7c234621231e3e3"
server_key_iv: "963d7a7cb3523e2cad52d6251a50edb1"
server_key_version: "encryptionkey@0"
I20260812 07:04:19.595597 13250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:19.609623 13250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:19.613513 13250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:19.613754 13250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:19.614705 13250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:19.616075 13250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:19.616151 13250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:19.616245 13250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:19.616293 13250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:19.651444 13250 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.2:46387
I20260812 07:04:19.651496 13397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.2:46387 every 8 connection(s)
I20260812 07:04:19.652827 13250 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
I20260812 07:04:19.662331 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 13250
I20260812 07:04:19.662532 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance
I20260812 07:04:19.663015 6388 external_mini_cluster.cc:1428] Setting key be5dd87ee95f52b5fde81e48381bc9c9
Aug 12 07:04:19 dist-test-slave-69rg krb5kdc[12965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518259, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:04:19.675594 13401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:19.655863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:37417 (local address 127.6.61.62:33245)
0812 07:04:19.656407 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:19.656412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:19.657368 (+ 956us) server_negotiation.cc:407] Connection header received
0812 07:04:19.659573 (+ 2205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:19.659579 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:19.659686 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:19.659820 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:19.661418 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:19.662228 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:19.664095 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:19.664791 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:19.664956 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:19.669611 (+ 4655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:19.669617 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:19.669621 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:19.669657 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:19.672283 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:19.673059 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:19.673065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:19.673068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:19.673203 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:19.673664 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:19.673669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:19.673672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:19.674032 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:19.674201 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:19.674968 (+ 767us) server_negotiation.cc:299] Negotiation successful
0812 07:04:19.675200 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":300,"threads_started":1}
I20260812 07:04:19.676611 13402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:19.656331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:33245 (local address 127.6.61.2:37417)
0812 07:04:19.657176 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:19.657241 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:19.659210 (+ 1969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:19.660080 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:19.660089 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:19.660639 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:19.661221 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:19.661239 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:19.662403 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:19.662428 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:19.663919 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:19.663935 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:19.665061 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:19.665070 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:19.665537 (+ 467us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:19.666450 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:19.666483 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:19.669336 (+ 2853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:19.672442 (+ 3106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:19.672452 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:19.672471 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:19.672928 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:19.673330 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:19.673335 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:19.673339 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:19.673545 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:19.674167 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:19.674176 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:19.674548 (+ 372us) client_negotiation.cc:769] Sending connection context
0812 07:04:19.674876 (+ 328us) client_negotiation.cc:240] Negotiation successful
0812 07:04:19.675447 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":342,"threads_started":1}
I20260812 07:04:19.678197 13399 heartbeater.cc:347] Connected to a master server at 127.6.61.62:33245
I20260812 07:04:19.678575 13399 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:19.679361 13399 heartbeater.cc:511] Master 127.6.61.62:33245 requested a full tablet report, sending...
I20260812 07:04:19.680980 13018 ts_manager.cc:194] Registered new tserver with Master: ec8cba52ec024bb3a945561a11ae03e0 (127.6.61.2:46387)
I20260812 07:04:19.682337 13018 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.2@KRBTEST.COM'} at 127.6.61.2:37417
WARNING: no policy specified for kudu/127.6.61.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
I20260812 07:04:19.722209 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.3:0
--local_ip_for_outbound_sockets=127.6.61.3
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:33245
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:38837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:20.073581 13406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:20.074059 13406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:20.074193 13406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:20.081933 13406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:20.082103 13406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:20.082373 13406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.3
I20260812 07:04:20.092787 13406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:38837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:33245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.3
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:20.094334 13406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:20.095717 13406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:20.110154 13411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:20.110088 13413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:20.110131 13415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:20.115706 13406 server_base.cc:1034] running on GCE node
I20260812 07:04:20.116498 13406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:20.117681 13406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:20.118891 13406 hybrid_clock.cc:648] HybridClock initialized: now 1786518260118847 us; error 33 us; skew 500 ppm
I20260812 07:04:20.119180 13406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:04:20 dist-test-slave-69rg krb5kdc[12965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518260, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:20.122587 13406 init.cc:377] Logged in from keytab as kudu/127.6.61.3@KRBTEST.COM (short username kudu)
I20260812 07:04:20.124061 13406 webserver.cc:460] Webserver started at http://127.6.61.3:36839/ using document root <none> and password file <none>
I20260812 07:04:20.124739 13406 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:20.124846 13406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:20.125119 13406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:20.128111 13406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-2/data/instance:
uuid: "2a734d442fdd4913b609198c3e92c04e"
format_stamp: "Formatted at 2026-08-12 07:04:20 on dist-test-slave-69rg"
server_key: "510795a2d542823c8220a1b81e975003"
server_key_iv: "37337b8fdfe5f19508f519abd69d206d"
server_key_version: "encryptionkey@0"
I20260812 07:04:20.128865 13406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance:
uuid: "2a734d442fdd4913b609198c3e92c04e"
format_stamp: "Formatted at 2026-08-12 07:04:20 on dist-test-slave-69rg"
server_key: "510795a2d542823c8220a1b81e975003"
server_key_iv: "37337b8fdfe5f19508f519abd69d206d"
server_key_version: "encryptionkey@0"
I20260812 07:04:20.134150 13406 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 07:04:20.137692 13422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:20.139153 13406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:04:20.139380 13406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-2/wal
uuid: "2a734d442fdd4913b609198c3e92c04e"
format_stamp: "Formatted at 2026-08-12 07:04:20 on dist-test-slave-69rg"
server_key: "510795a2d542823c8220a1b81e975003"
server_key_iv: "37337b8fdfe5f19508f519abd69d206d"
server_key_version: "encryptionkey@0"
I20260812 07:04:20.139599 13406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:20.148581 13243 heartbeater.cc:503] Master 127.6.61.62:33245 was elected leader, sending a full tablet report...
I20260812 07:04:20.169260 13406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:20.174142 13406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:20.174466 13406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:20.175521 13406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:20.177176 13406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:20.177291 13406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:20.177487 13406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:20.177572 13406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:20.214288 13406 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.3:44363
I20260812 07:04:20.214342 13552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.3:44363 every 8 connection(s)
I20260812 07:04:20.215624 13406 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
I20260812 07:04:20.218001 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 13406
I20260812 07:04:20.218221 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance
I20260812 07:04:20.218858 6388 external_mini_cluster.cc:1428] Setting key 7b2dbf88ff68a816a80a8b9234bd7a29
Aug 12 07:04:20 dist-test-slave-69rg krb5kdc[12965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518260, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:04:20.236721 13555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:20.218457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:32933 (local address 127.6.61.62:33245)
0812 07:04:20.219064 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:20.219071 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:20.220082 (+ 1011us) server_negotiation.cc:407] Connection header received
0812 07:04:20.222205 (+ 2123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:20.222209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:20.222302 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:20.222392 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:20.223853 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:20.224662 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:20.226294 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:20.226689 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:20.226786 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:20.230397 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:20.230403 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:20.230407 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:20.230439 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:20.233420 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:20.234431 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:20.234435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:20.234437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:20.234617 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:20.235095 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:20.235098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:20.235100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:20.235424 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:20.235538 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:20.236240 (+ 702us) server_negotiation.cc:299] Negotiation successful
0812 07:04:20.236449 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":269,"threads_started":1}
I20260812 07:04:20.237704 13556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:20.219048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:33245 (local address 127.6.61.3:32933)
0812 07:04:20.219934 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:20.219980 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:20.221944 (+ 1964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:20.222593 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:20.222600 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:20.223134 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:20.223684 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:20.223700 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:20.224815 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:20.224819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:20.226148 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:20.226161 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:20.226893 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:20.226898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:20.227085 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:20.227801 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:20.227823 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:20.230180 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:20.233566 (+ 3386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:20.233575 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:20.233593 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:20.234266 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:20.234754 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:20.234759 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:20.234762 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:20.234987 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:20.235540 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:20.235546 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:20.235885 (+ 339us) client_negotiation.cc:769] Sending connection context
0812 07:04:20.236135 (+ 250us) client_negotiation.cc:240] Negotiation successful
0812 07:04:20.236548 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":396,"threads_started":1}
I20260812 07:04:20.239230 13553 heartbeater.cc:347] Connected to a master server at 127.6.61.62:33245
I20260812 07:04:20.239519 13553 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:20.240262 13553 heartbeater.cc:511] Master 127.6.61.62:33245 requested a full tablet report, sending...
I20260812 07:04:20.241729 13018 ts_manager.cc:194] Registered new tserver with Master: 2a734d442fdd4913b609198c3e92c04e (127.6.61.3:44363)
I20260812 07:04:20.242641 13018 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.3@KRBTEST.COM'} at 127.6.61.3:32933
I20260812 07:04:20.247152 6388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/test-admin.keytab.
Aug 12 07:04:20 dist-test-slave-69rg krb5kdc[12965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:20.280305 6388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260812 07:04:20.286417 13562 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:20.283535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60952 (local address 127.6.61.1:36155)
0812 07:04:20.284328 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:20.284336 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:20.284364 (+ 28us) server_negotiation.cc:407] Connection header received
0812 07:04:20.284428 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:20.284434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:20.284649 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:20.284780 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:20.285997 (+ 1217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60952: BlockingRecv error: recv got EOF from 127.0.0.1:60952 (error 108)
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":447,"threads_started":1}
W20260812 07:04:20.286607 6388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/krb5cc' of type FILE
Aug 12 07:04:20 dist-test-slave-69rg krb5kdc[12965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:20.288729 6388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 07:04:20 dist-test-slave-69rg krb5kdc[12965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:20.290269 6388 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 07:04:20 dist-test-slave-69rg krb5kdc[12965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.1@KRBTEST.COM
I20260812 07:04:20.307709 13562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:20.292162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60958 (local address 127.6.61.1:36155)
0812 07:04:20.292422 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:20.292429 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:20.292602 (+ 173us) server_negotiation.cc:407] Connection header received
0812 07:04:20.292828 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:20.292833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:20.292897 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:20.293004 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:20.294164 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:20.295161 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:20.296429 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:20.297020 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:20.297168 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:20.301192 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:20.301205 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:20.301222 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:20.301265 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:20.304554 (+ 3289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:20.305457 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:20.305465 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:20.305467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:20.305624 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:20.306075 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:20.306082 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:20.306085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:20.306484 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:20.306812 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:20.307141 (+ 329us) server_negotiation.cc:299] Negotiation successful
0812 07:04:20.307431 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260812 07:04:20.314164 13562 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:20.312159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60970 (local address 127.6.61.1:36155)
0812 07:04:20.312439 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:20.312445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:20.312696 (+ 251us) server_negotiation.cc:407] Connection header received
0812 07:04:20.312841 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:20.312844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:20.312917 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:20.313001 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:20.313900 (+ 899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60970: BlockingRecv error: recv got EOF from 127.0.0.1:60970 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260812 07:04:20.314370 6388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/krb5cc' of type FILE
Aug 12 07:04:20 dist-test-slave-69rg krb5kdc[12965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:20.316139 6388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 07:04:20 dist-test-slave-69rg krb5kdc[12965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:20.317799 6388 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 07:04:20 dist-test-slave-69rg krb5kdc[12965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.1@KRBTEST.COM
I20260812 07:04:20.334164 13562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:20.319559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60978 (local address 127.6.61.1:36155)
0812 07:04:20.319836 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:20.319844 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:20.319994 (+ 150us) server_negotiation.cc:407] Connection header received
0812 07:04:20.320133 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:20.320138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:20.320207 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:20.320309 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:20.321436 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:20.322327 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:20.323721 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:20.324198 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:20.324326 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:20.328233 (+ 3907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:20.328238 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:20.328242 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:20.328295 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:20.331235 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:20.331961 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:20.331969 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:20.331973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:20.332125 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:20.332565 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:20.332571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:20.332574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:20.332887 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:20.333187 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:20.333514 (+ 327us) server_negotiation.cc:299] Negotiation successful
0812 07:04:20.333757 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
W20260812 07:04:20.338727 13562 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:20.336904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60992 (local address 127.6.61.1:36155)
0812 07:04:20.337138 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:20.337145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:20.337401 (+ 256us) server_negotiation.cc:407] Connection header received
0812 07:04:20.337526 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:20.337530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:20.337604 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:20.337687 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:20.338491 (+ 804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60992: BlockingRecv error: recv got EOF from 127.0.0.1:60992 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260812 07:04:20.338912 6388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518208686283-6388-0/krb5kdc/krb5cc' of type FILE
Aug 12 07:04:20 dist-test-slave-69rg krb5kdc[12965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:20.340479 6388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 07:04:20 dist-test-slave-69rg krb5kdc[12965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:20.342116 6388 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 07:04:20 dist-test-slave-69rg krb5kdc[12965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.1@KRBTEST.COM
I20260812 07:04:20.358044 13562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:20.343567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60998 (local address 127.6.61.1:36155)
0812 07:04:20.343821 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:20.343826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:20.344102 (+ 276us) server_negotiation.cc:407] Connection header received
0812 07:04:20.344304 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:20.344308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:20.344372 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:20.344459 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:20.345440 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:20.346328 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:20.347598 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:20.348065 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:20.348294 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:20.352220 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:20.352225 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:20.352229 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:20.352267 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:20.355090 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:20.355847 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:20.355854 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:20.355859 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:20.355997 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:20.356388 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:20.356393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:20.356396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:20.356706 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:20.357071 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:20.357352 (+ 281us) server_negotiation.cc:299] Negotiation successful
0812 07:04:20.357590 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260812 07:04:20.359961 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 13093
I20260812 07:04:20.373150 13562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:20.360766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32768 (local address 127.6.61.1:36155)
0812 07:04:20.361005 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:20.361013 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:20.361297 (+ 284us) server_negotiation.cc:407] Connection header received
0812 07:04:20.361407 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:20.361411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:20.361472 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:20.361555 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:20.362612 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:20.363404 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:20.364648 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:20.365124 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:20.365248 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:20.367663 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:20.367669 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:20.367672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:20.367706 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:20.370662 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:20.371337 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:20.371342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:20.371344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:20.371494 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:20.371826 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:20.371830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:20.371832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:20.372102 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:20.372314 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:20.372568 (+ 254us) server_negotiation.cc:299] Negotiation successful
0812 07:04:20.372744 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 07:04:20.373986 13237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:20.458668 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 13093
I20260812 07:04:20.481225 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 13250
Aug 12 07:04:20 dist-test-slave-69rg krb5kdc[12965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.2@KRBTEST.COM
I20260812 07:04:20.497408 13577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:20.482254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45404 (local address 127.6.61.2:46387)
0812 07:04:20.482813 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:20.482821 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:20.482849 (+ 28us) server_negotiation.cc:407] Connection header received
0812 07:04:20.482910 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:20.482917 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:20.483122 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:20.483256 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:20.484328 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:20.485369 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:20.486887 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:20.487412 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:20.487515 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:20.491458 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:20.491471 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:20.491485 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:20.491521 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:20.494425 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:20.495129 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:20.495136 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:20.495139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:20.495274 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:20.495658 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:20.495661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:20.495664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:20.496053 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:20.496298 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:20.496867 (+ 569us) server_negotiation.cc:299] Negotiation successful
0812 07:04:20.497078 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":282,"threads_started":1}
I20260812 07:04:20.498778 13392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:20.587116 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 13250
I20260812 07:04:20.607882 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 13406
Aug 12 07:04:20 dist-test-slave-69rg krb5kdc[12965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.3@KRBTEST.COM
I20260812 07:04:20.624570 13580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:20.609161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47002 (local address 127.6.61.3:44363)
0812 07:04:20.610013 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:20.610021 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:20.610054 (+ 33us) server_negotiation.cc:407] Connection header received
0812 07:04:20.610122 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:20.610130 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:20.610386 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:20.610573 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:20.611739 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:20.612760 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:20.614082 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:20.614643 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:20.614781 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:20.618886 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:20.618898 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:20.618912 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:20.618947 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:20.621707 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:20.622425 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:20.622441 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:20.622443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:20.622565 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:20.622983 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:20.622987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:20.622990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:20.623347 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:20.623682 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:20.623993 (+ 311us) server_negotiation.cc:299] Negotiation successful
0812 07:04:20.624266 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":345,"threads_started":1}
I20260812 07:04:20.626034 13548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:20.712445 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 13406
I20260812 07:04:20.735074 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 12981
I20260812 07:04:20.736526 13053 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:20.841579 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 12981
2026-08-12T07:04:20Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4260 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:04:20 dist-test-slave-69rg krb5kdc[13585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:04:20 dist-test-slave-69rg krb5kdc[13585](info): set up 2 sockets
Aug 12 07:04:20 dist-test-slave-69rg krb5kdc[13585](info): commencing operation
krb5kdc: starting...
W20260812 07:04:22.054915 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.166s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:04:22 dist-test-slave-69rg krb5kdc[13585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:04:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:04:22Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.62.keytab.
Entry for principal oryx/127.6.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.62.keytab.
I20260812 07:04:22.314451 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:34449
--webserver_interface=127.6.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.62.keytab
--principal=oryx/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:34261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:34449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:22.650624 13600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:22.650918 13600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:22.651101 13600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:22.658464 13600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:04:22.658576 13600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:22.658661 13600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:22.658751 13600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:04:22.658906 13600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:04:22.668949 13600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:34261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:34449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.62.keytab
--principal=oryx/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:34449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:22.670363 13600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:22.671729 13600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:22.683991 13606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:22.684051 13609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:22.684056 13607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:22.685204 13600 server_base.cc:1034] running on GCE node
I20260812 07:04:22.686120 13600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:22.687623 13600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:22.688859 13600 hybrid_clock.cc:648] HybridClock initialized: now 1786518262688785 us; error 55 us; skew 500 ppm
I20260812 07:04:22.689206 13600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:04:22 dist-test-slave-69rg krb5kdc[13585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518262, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:22.692658 13600 init.cc:377] Logged in from keytab as oryx/127.6.61.62@KRBTEST.COM (short username oryx)
I20260812 07:04:22.694187 13600 webserver.cc:460] Webserver started at http://127.6.61.62:46129/ using document root <none> and password file <none>
I20260812 07:04:22.694885 13600 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:22.695024 13600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:22.695385 13600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:22.698499 13600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "6c86051a8e4146a3b059183c03351248"
format_stamp: "Formatted at 2026-08-12 07:04:22 on dist-test-slave-69rg"
server_key: "4573d473989d50b00412dc2db8594858"
server_key_iv: "b5fbbf1cbb521024ece2284eac95c3fe"
server_key_version: "encryptionkey@0"
I20260812 07:04:22.699285 13600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "6c86051a8e4146a3b059183c03351248"
format_stamp: "Formatted at 2026-08-12 07:04:22 on dist-test-slave-69rg"
server_key: "4573d473989d50b00412dc2db8594858"
server_key_iv: "b5fbbf1cbb521024ece2284eac95c3fe"
server_key_version: "encryptionkey@0"
I20260812 07:04:22.704375 13600 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 07:04:22.707868 13617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:22.709219 13600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 07:04:22.709405 13600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "6c86051a8e4146a3b059183c03351248"
format_stamp: "Formatted at 2026-08-12 07:04:22 on dist-test-slave-69rg"
server_key: "4573d473989d50b00412dc2db8594858"
server_key_iv: "b5fbbf1cbb521024ece2284eac95c3fe"
server_key_version: "encryptionkey@0"
I20260812 07:04:22.709604 13600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:22.751305 13600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:22.755431 13600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:22.755736 13600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:22.774906 13600 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:34449
I20260812 07:04:22.774969 13678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:34449 every 8 connection(s)
I20260812 07:04:22.776377 13600 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:04:22.780340 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 13600
I20260812 07:04:22.780570 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:04:22.781472 6388 external_mini_cluster.cc:1428] Setting key 6f59fe59b2b77a9a2e38f60792736272
I20260812 07:04:22.781481 13679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:22.794687 13679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248: Bootstrap starting.
Aug 12 07:04:22 dist-test-slave-69rg krb5kdc[13585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM
I20260812 07:04:22.798154 13679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:22.799400 13679 log.cc:824] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:22.802397 13679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248: No bootstrap required, opened a new log
I20260812 07:04:22.804473 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:22.783825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60020 (local address 127.6.61.62:34449)
0812 07:04:22.784667 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:22.784679 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:22.784723 (+ 44us) server_negotiation.cc:407] Connection header received
0812 07:04:22.786565 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:22.786587 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:22.787054 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:22.787449 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:22.789044 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:22.790431 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:22.791799 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:22.792493 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:22.792725 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:22.797088 (+ 4363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:22.797096 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:22.797110 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:22.797147 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:22.800351 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:22.801065 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:22.801071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:22.801078 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:22.801247 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:22.801669 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:22.801674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:22.801677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:22.802265 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:22.802598 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:22.803060 (+ 462us) server_negotiation.cc:299] Negotiation successful
0812 07:04:22.803484 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":386,"threads_started":1}
I20260812 07:04:22.810482 13679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } }
I20260812 07:04:22.810979 13679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:22.811100 13679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c86051a8e4146a3b059183c03351248, State: Initialized, Role: FOLLOWER
I20260812 07:04:22.811666 13679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } }
I20260812 07:04:22.811911 13679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:04:22.812074 13679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:04:22.812254 13679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:22.814780 13679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } }
I20260812 07:04:22.815289 13679 leader_election.cc:302] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c86051a8e4146a3b059183c03351248; no voters:
I20260812 07:04:22.815924 13679 leader_election.cc:288] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:04:22.816051 13684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:04:22.816771 13684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [term 1 LEADER]: Becoming Leader. State: Replica: 6c86051a8e4146a3b059183c03351248, State: Running, Role: LEADER
I20260812 07:04:22.817373 13684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } }
I20260812 07:04:22.818198 13679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:04:22.820132 13685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c86051a8e4146a3b059183c03351248" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } }
I20260812 07:04:22.820287 13686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c86051a8e4146a3b059183c03351248. Latest consensus state: current_term: 1 leader_uuid: "6c86051a8e4146a3b059183c03351248" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } }
I20260812 07:04:22.820533 13685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:22.820564 13686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:22.821372 13691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:04:22.826982 13691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:04:22.838891 13691 catalog_manager.cc:1347] Generated new cluster ID: 71d35a330bec42eb92f085ce763b979e
I20260812 07:04:22.839053 13691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:04:22.864163 13691 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:04:22.865638 13691 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:04:22.877727 13691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248: Generated new TSK 0
I20260812 07:04:22.878999 13691 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.1.keytab.
Entry for principal oryx/127.6.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.1.keytab.
I20260812 07:04:22.957610 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.1:0
--local_ip_for_outbound_sockets=127.6.61.1
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:34449
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.1.keytab
--principal=oryx/127.6.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:34261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:23.307186 13709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:23.307482 13709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:23.307639 13709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:23.315295 13709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:23.315414 13709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:23.315626 13709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.1
I20260812 07:04:23.326159 13709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:34261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.1.keytab
--principal=oryx/127.6.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:34449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.1
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:23.327498 13709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:23.328856 13709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:23.342484 13715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:23.342555 13714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:23.342547 13718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:23.343475 13709 server_base.cc:1034] running on GCE node
I20260812 07:04:23.344099 13709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:23.345126 13709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:23.346390 13709 hybrid_clock.cc:648] HybridClock initialized: now 1786518263346343 us; error 88 us; skew 500 ppm
I20260812 07:04:23.346740 13709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:04:23 dist-test-slave-69rg krb5kdc[13585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518263, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:23.350137 13709 init.cc:377] Logged in from keytab as oryx/127.6.61.1@KRBTEST.COM (short username oryx)
I20260812 07:04:23.351576 13709 webserver.cc:460] Webserver started at http://127.6.61.1:44895/ using document root <none> and password file <none>
I20260812 07:04:23.352293 13709 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:23.352453 13709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:23.352767 13709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:23.355846 13709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-0/data/instance:
uuid: "30fd1f2ae015406199f6b131c77b167f"
format_stamp: "Formatted at 2026-08-12 07:04:23 on dist-test-slave-69rg"
server_key: "abd074051d2d8197d0bdcee0b280ae92"
server_key_iv: "842ac178059b35faf287422b8c2d6889"
server_key_version: "encryptionkey@0"
I20260812 07:04:23.356628 13709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance:
uuid: "30fd1f2ae015406199f6b131c77b167f"
format_stamp: "Formatted at 2026-08-12 07:04:23 on dist-test-slave-69rg"
server_key: "abd074051d2d8197d0bdcee0b280ae92"
server_key_iv: "842ac178059b35faf287422b8c2d6889"
server_key_version: "encryptionkey@0"
I20260812 07:04:23.361732 13709 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260812 07:04:23.365347 13726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:23.366894 13709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 07:04:23.367086 13709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-0/wal
uuid: "30fd1f2ae015406199f6b131c77b167f"
format_stamp: "Formatted at 2026-08-12 07:04:23 on dist-test-slave-69rg"
server_key: "abd074051d2d8197d0bdcee0b280ae92"
server_key_iv: "842ac178059b35faf287422b8c2d6889"
server_key_version: "encryptionkey@0"
I20260812 07:04:23.367312 13709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:23.384826 13709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:23.389181 13709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:23.389484 13709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:23.390554 13709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:23.391947 13709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:23.392053 13709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:23.392233 13709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:23.392292 13709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:23.426952 13709 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.1:42623
I20260812 07:04:23.426996 13856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.1:42623 every 8 connection(s)
I20260812 07:04:23.428328 13709 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
I20260812 07:04:23.436681 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 13709
I20260812 07:04:23.436939 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance
I20260812 07:04:23.437862 6388 external_mini_cluster.cc:1428] Setting key 81fa5e2f3707abbdfa97e4ca98aa84b8
Aug 12 07:04:23 dist-test-slave-69rg krb5kdc[13585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518263, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM
I20260812 07:04:23.455044 13859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:23.431071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:59467 (local address 127.6.61.62:34449)
0812 07:04:23.431525 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:23.431530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:23.432495 (+ 965us) server_negotiation.cc:407] Connection header received
0812 07:04:23.434474 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:23.434479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:23.434549 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:23.434646 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:23.436065 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:23.436805 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:23.439023 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:23.439461 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:23.439560 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:23.443283 (+ 3723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:23.443289 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:23.443292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:23.443327 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:23.451251 (+ 7924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:23.452321 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:23.452326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:23.452328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:23.452455 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:23.453049 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:23.453053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:23.453056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:23.453397 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:23.453523 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:23.454579 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0812 07:04:23.454761 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":346,"thread_start_us":234,"threads_started":1}
I20260812 07:04:23.456632 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:23.431606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.6.61.1:59467)
0812 07:04:23.432338 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:23.432385 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:23.434224 (+ 1839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:23.434874 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:23.434882 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:23.435359 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:23.435910 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:23.435927 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:23.437008 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:23.437017 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:23.438870 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:23.438883 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:23.439699 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:23.439706 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:23.440028 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:23.440731 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:23.440754 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:23.443048 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:23.451433 (+ 8385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:23.451445 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:23.451468 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:23.452144 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:23.452577 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:23.452584 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:23.452590 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:23.452860 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:23.453545 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:23.453557 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:23.454106 (+ 549us) client_negotiation.cc:769] Sending connection context
0812 07:04:23.454457 (+ 351us) client_negotiation.cc:240] Negotiation successful
0812 07:04:23.455060 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":258,"threads_started":1}
I20260812 07:04:23.458498 13857 heartbeater.cc:347] Connected to a master server at 127.6.61.62:34449
I20260812 07:04:23.458834 13857 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:23.459595 13857 heartbeater.cc:511] Master 127.6.61.62:34449 requested a full tablet report, sending...
I20260812 07:04:23.462188 13639 ts_manager.cc:194] Registered new tserver with Master: 30fd1f2ae015406199f6b131c77b167f (127.6.61.1:42623)
I20260812 07:04:23.464447 13639 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.61.1@KRBTEST.COM'} at 127.6.61.1:59467
WARNING: no policy specified for oryx/127.6.61.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.61.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.2.keytab.
Entry for principal oryx/127.6.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.2.keytab.
I20260812 07:04:23.495546 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.2:0
--local_ip_for_outbound_sockets=127.6.61.2
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:34449
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.2.keytab
--principal=oryx/127.6.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:34261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:23.836759 13864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:23.837070 13864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:23.837170 13864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:23.844919 13864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:23.845065 13864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:23.845278 13864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.2
I20260812 07:04:23.855594 13864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:34261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.2.keytab
--principal=oryx/127.6.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:34449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.2
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:23.856984 13864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:23.858327 13864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:23.872452 13873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:23.872553 13871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:23.872490 13870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:23.873327 13864 server_base.cc:1034] running on GCE node
I20260812 07:04:23.874027 13864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:23.875015 13864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:23.876263 13864 hybrid_clock.cc:648] HybridClock initialized: now 1786518263876224 us; error 52 us; skew 500 ppm
I20260812 07:04:23.876603 13864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:04:23 dist-test-slave-69rg krb5kdc[13585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518263, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:23.880163 13864 init.cc:377] Logged in from keytab as oryx/127.6.61.2@KRBTEST.COM (short username oryx)
I20260812 07:04:23.881719 13864 webserver.cc:460] Webserver started at http://127.6.61.2:39463/ using document root <none> and password file <none>
I20260812 07:04:23.882467 13864 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:23.882573 13864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:23.882848 13864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:23.886029 13864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-1/data/instance:
uuid: "d6f37996431d4b60afeb49f20e3fbba4"
format_stamp: "Formatted at 2026-08-12 07:04:23 on dist-test-slave-69rg"
server_key: "bf17441fb24d90c969a0e5a1b909e697"
server_key_iv: "eec2a9197c0fa72a96d9116a95fffd71"
server_key_version: "encryptionkey@0"
I20260812 07:04:23.886793 13864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance:
uuid: "d6f37996431d4b60afeb49f20e3fbba4"
format_stamp: "Formatted at 2026-08-12 07:04:23 on dist-test-slave-69rg"
server_key: "bf17441fb24d90c969a0e5a1b909e697"
server_key_iv: "eec2a9197c0fa72a96d9116a95fffd71"
server_key_version: "encryptionkey@0"
I20260812 07:04:23.892684 13864 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260812 07:04:23.896422 13881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:23.897751 13864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 07:04:23.897987 13864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-1/wal
uuid: "d6f37996431d4b60afeb49f20e3fbba4"
format_stamp: "Formatted at 2026-08-12 07:04:23 on dist-test-slave-69rg"
server_key: "bf17441fb24d90c969a0e5a1b909e697"
server_key_iv: "eec2a9197c0fa72a96d9116a95fffd71"
server_key_version: "encryptionkey@0"
I20260812 07:04:23.898187 13864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:23.921095 13864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:23.925073 13864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:23.925321 13864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:23.926318 13864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:23.927677 13864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:23.927779 13864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:23.927868 13864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:23.927914 13864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:23.962275 13864 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.2:39417
I20260812 07:04:23.962371 14016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.2:39417 every 8 connection(s)
I20260812 07:04:23.963605 13864 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
I20260812 07:04:23.970897 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 13864
I20260812 07:04:23.971113 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance
I20260812 07:04:23.971663 6388 external_mini_cluster.cc:1428] Setting key 953d6e359867bae3438acf8b9323ccbd
Aug 12 07:04:23 dist-test-slave-69rg krb5kdc[13585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518263, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM
I20260812 07:04:23.984858 14019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:23.966374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:35425 (local address 127.6.61.62:34449)
0812 07:04:23.966793 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:23.966798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:23.967754 (+ 956us) server_negotiation.cc:407] Connection header received
0812 07:04:23.969630 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:23.969637 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:23.969737 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:23.969894 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:23.971707 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:23.972573 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:23.974220 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:23.974927 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:23.975103 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:23.978842 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:23.978847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:23.978850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:23.978885 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:23.981533 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:23.982402 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:23.982408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:23.982410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:23.982534 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:23.982999 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:23.983003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:23.983005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:23.983296 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:23.983469 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:23.984218 (+ 749us) server_negotiation.cc:299] Negotiation successful
0812 07:04:23.984486 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":201,"threads_started":1}
I20260812 07:04:23.985942 14020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:23.966889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.6.61.2:35425)
0812 07:04:23.967587 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:23.967629 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:23.969347 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:23.970127 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:23.970135 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:23.970736 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:23.971508 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:23.971536 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:23.972737 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:23.972745 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:23.974073 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:23.974086 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:23.975194 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:23.975203 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:23.975450 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:23.976175 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:23.976198 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:23.978589 (+ 2391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:23.981703 (+ 3114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:23.981713 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:23.981731 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:23.982249 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:23.982652 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:23.982657 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:23.982661 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:23.982866 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:23.983443 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:23.983451 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:23.983822 (+ 371us) client_negotiation.cc:769] Sending connection context
0812 07:04:23.984100 (+ 278us) client_negotiation.cc:240] Negotiation successful
0812 07:04:23.984661 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":268,"threads_started":1}
I20260812 07:04:23.987509 14017 heartbeater.cc:347] Connected to a master server at 127.6.61.62:34449
I20260812 07:04:23.987877 14017 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:23.988788 14017 heartbeater.cc:511] Master 127.6.61.62:34449 requested a full tablet report, sending...
I20260812 07:04:23.990658 13639 ts_manager.cc:194] Registered new tserver with Master: d6f37996431d4b60afeb49f20e3fbba4 (127.6.61.2:39417)
I20260812 07:04:23.992000 13639 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.61.2@KRBTEST.COM'} at 127.6.61.2:35425
WARNING: no policy specified for oryx/127.6.61.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.61.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.3.keytab.
Entry for principal oryx/127.6.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.3.keytab.
I20260812 07:04:24.034070 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.3:0
--local_ip_for_outbound_sockets=127.6.61.3
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:34449
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.3.keytab
--principal=oryx/127.6.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:34261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:24.377717 14023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:24.378228 14023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:24.378422 14023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:24.386786 14023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:24.386955 14023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:24.387148 14023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.3
I20260812 07:04:24.397782 14023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:34261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.3.keytab
--principal=oryx/127.6.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:34449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.3
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:24.399188 14023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:24.400537 14023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:24.414296 14029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:24.414444 14030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:24.414690 14033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:24.415925 14023 server_base.cc:1034] running on GCE node
I20260812 07:04:24.416695 14023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:24.417704 14023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:24.418927 14023 hybrid_clock.cc:648] HybridClock initialized: now 1786518264418826 us; error 88 us; skew 500 ppm
I20260812 07:04:24.419250 14023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:04:24 dist-test-slave-69rg krb5kdc[13585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518264, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:24.422791 14023 init.cc:377] Logged in from keytab as oryx/127.6.61.3@KRBTEST.COM (short username oryx)
I20260812 07:04:24.424405 14023 webserver.cc:460] Webserver started at http://127.6.61.3:38245/ using document root <none> and password file <none>
I20260812 07:04:24.425143 14023 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:24.425297 14023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:24.425612 14023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:24.428648 14023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-2/data/instance:
uuid: "1a5f51df7c0c43bc85366d1c15dc2f89"
format_stamp: "Formatted at 2026-08-12 07:04:24 on dist-test-slave-69rg"
server_key: "648e331102b7840000c5bfdb606abd59"
server_key_iv: "dc7192be850187f577207fe7bf7b9397"
server_key_version: "encryptionkey@0"
I20260812 07:04:24.429441 14023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance:
uuid: "1a5f51df7c0c43bc85366d1c15dc2f89"
format_stamp: "Formatted at 2026-08-12 07:04:24 on dist-test-slave-69rg"
server_key: "648e331102b7840000c5bfdb606abd59"
server_key_iv: "dc7192be850187f577207fe7bf7b9397"
server_key_version: "encryptionkey@0"
I20260812 07:04:24.434737 14023 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 07:04:24.438191 14040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:24.439507 14023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 07:04:24.439703 14023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-2/wal
uuid: "1a5f51df7c0c43bc85366d1c15dc2f89"
format_stamp: "Formatted at 2026-08-12 07:04:24 on dist-test-slave-69rg"
server_key: "648e331102b7840000c5bfdb606abd59"
server_key_iv: "dc7192be850187f577207fe7bf7b9397"
server_key_version: "encryptionkey@0"
I20260812 07:04:24.439909 14023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:24.459936 14023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:24.464327 14023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:24.464654 14023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:24.465787 14023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:24.467518 13857 heartbeater.cc:503] Master 127.6.61.62:34449 was elected leader, sending a full tablet report...
I20260812 07:04:24.467717 14023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:24.467844 14023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:24.468019 14023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:24.468091 14023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:24.506392 14023 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.3:34313
I20260812 07:04:24.506471 14170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.3:34313 every 8 connection(s)
I20260812 07:04:24.507699 14023 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
I20260812 07:04:24.509354 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 14023
I20260812 07:04:24.509555 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance
I20260812 07:04:24.510160 6388 external_mini_cluster.cc:1428] Setting key 4ea4193b289dae2a2aef95f14a409773
Aug 12 07:04:24 dist-test-slave-69rg krb5kdc[13585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518264, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM
I20260812 07:04:24.529533 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:24.510637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:32955 (local address 127.6.61.62:34449)
0812 07:04:24.511167 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:24.511173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:24.512529 (+ 1356us) server_negotiation.cc:407] Connection header received
0812 07:04:24.514632 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:24.514636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:24.514701 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:24.514809 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:24.516282 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:24.517063 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:24.518832 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:24.519305 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:24.519404 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:24.523009 (+ 3605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:24.523014 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:24.523017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:24.523050 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:24.526352 (+ 3302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:24.527124 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:24.527129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:24.527135 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:24.527261 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:24.527706 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:24.527711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:24.527716 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:24.527991 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:24.528159 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:24.528894 (+ 735us) server_negotiation.cc:299] Negotiation successful
0812 07:04:24.529136 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":281,"threads_started":1}
I20260812 07:04:24.530530 14174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:24.511142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.6.61.3:32955)
0812 07:04:24.512156 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:24.512215 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:24.514415 (+ 2200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:24.515007 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:24.515015 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:24.515539 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:24.516118 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:24.516138 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:24.517199 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:24.517203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:24.518697 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:24.518711 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:24.519522 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:24.519531 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:24.519751 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:24.520457 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:24.520484 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:24.522791 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:24.526511 (+ 3720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:24.526519 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:24.526533 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:24.526982 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:24.527377 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:24.527382 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:24.527390 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:24.527578 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:24.528159 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:24.528168 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:24.528503 (+ 335us) client_negotiation.cc:769] Sending connection context
0812 07:04:24.528775 (+ 272us) client_negotiation.cc:240] Negotiation successful
0812 07:04:24.529319 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"thread_start_us":489,"threads_started":1}
I20260812 07:04:24.531961 14171 heartbeater.cc:347] Connected to a master server at 127.6.61.62:34449
I20260812 07:04:24.532418 14171 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:24.533211 14171 heartbeater.cc:511] Master 127.6.61.62:34449 requested a full tablet report, sending...
I20260812 07:04:24.534698 13639 ts_manager.cc:194] Registered new tserver with Master: 1a5f51df7c0c43bc85366d1c15dc2f89 (127.6.61.3:34313)
I20260812 07:04:24.535488 13639 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.61.3@KRBTEST.COM'} at 127.6.61.3:32955
I20260812 07:04:24.539043 6388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.61.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.61.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.61.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.61.keytab.
Entry for principal oryx/127.6.61.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.61.keytab.
I20260812 07:04:24.595643 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.61:39641
--webserver_interface=127.6.61.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.61.keytab
--principal=oryx/127.6.61.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:34261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:34449,127.6.61.61:39641
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:24.950970 14179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:24.951259 14179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:24.951365 14179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:24.958755 14179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:04:24.958910 14179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:24.958994 14179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:24.959057 14179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:04:24.959105 14179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:04:24.968967 14179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:34261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:34449,127.6.61.61:39641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.61.keytab
--principal=oryx/127.6.61.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.61:39641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.61.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
W20260812 07:04:24.969357 14179 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.6.61.62:34449,127.6.61.61:39641'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260812 07:04:24.970379 14179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:24.971765 14179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:24.984386 14185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:24.984336 14184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:24.984361 14187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:24.985260 14179 server_base.cc:1034] running on GCE node
I20260812 07:04:24.986166 14179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:24.987725 14179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:24.988976 14179 hybrid_clock.cc:648] HybridClock initialized: now 1786518264988899 us; error 53 us; skew 500 ppm
I20260812 07:04:24.989338 14179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:04:24 dist-test-slave-69rg krb5kdc[13585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518264, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:24.992704 14179 init.cc:377] Logged in from keytab as oryx/127.6.61.61@KRBTEST.COM (short username oryx)
I20260812 07:04:24.994357 14179 webserver.cc:460] Webserver started at http://127.6.61.61:33019/ using document root <none> and password file <none>
I20260812 07:04:24.995066 14179 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:24.995206 14179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:24.995536 14179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:24.995779 14017 heartbeater.cc:503] Master 127.6.61.62:34449 was elected leader, sending a full tablet report...
I20260812 07:04:24.999536 14179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-1/data/instance:
uuid: "06a305d836de4590a62f589f803f0dfa"
format_stamp: "Formatted at 2026-08-12 07:04:24 on dist-test-slave-69rg"
server_key: "8a6c7fbeeefdecc14c21c3d9cb9342c7"
server_key_iv: "597105f727623042f2cb517fee2b6e78"
server_key_version: "encryptionkey@0"
I20260812 07:04:25.000391 14179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-1/wal/instance:
uuid: "06a305d836de4590a62f589f803f0dfa"
format_stamp: "Formatted at 2026-08-12 07:04:24 on dist-test-slave-69rg"
server_key: "8a6c7fbeeefdecc14c21c3d9cb9342c7"
server_key_iv: "597105f727623042f2cb517fee2b6e78"
server_key_version: "encryptionkey@0"
I20260812 07:04:25.005951 14179 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260812 07:04:25.009452 14194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:25.011001 14179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:04:25.011196 14179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-1/wal
uuid: "06a305d836de4590a62f589f803f0dfa"
format_stamp: "Formatted at 2026-08-12 07:04:24 on dist-test-slave-69rg"
server_key: "8a6c7fbeeefdecc14c21c3d9cb9342c7"
server_key_iv: "597105f727623042f2cb517fee2b6e78"
server_key_version: "encryptionkey@0"
I20260812 07:04:25.011418 14179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:25.033607 14179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:25.038290 14179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:25.038570 14179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:25.058552 14179 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.61:39641
I20260812 07:04:25.058609 14251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.61:39641 every 8 connection(s)
I20260812 07:04:25.060133 14179 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-1/data/info.pb
I20260812 07:04:25.061331 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 14179
I20260812 07:04:25.064587 14252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:25.073323 14252 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 07:04:25 dist-test-slave-69rg krb5kdc[13585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.61.61@KRBTEST.COM
Aug 12 07:04:25 dist-test-slave-69rg krb5kdc[13585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518264, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.61@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM
I20260812 07:04:25.087929 14258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:25.063865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60720 (local address 127.6.61.61:39641)
0812 07:04:25.064773 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:25.064786 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:25.064829 (+ 43us) server_negotiation.cc:407] Connection header received
0812 07:04:25.066630 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:25.066654 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:25.067042 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:25.067423 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:25.069397 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.070917 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.072758 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.073452 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.073602 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:25.078934 (+ 5332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:25.078945 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:25.078957 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:25.079011 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:25.083007 (+ 3996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:25.083862 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:25.083868 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:25.083874 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:25.084033 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:25.084528 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:25.084539 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:25.084542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:25.085323 (+ 781us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:25.085615 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:25.086127 (+ 512us) server_negotiation.cc:299] Negotiation successful
0812 07:04:25.086648 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":427,"threads_started":1}
I20260812 07:04:25.094058 14259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:25.076616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60032 (local address 127.6.61.62:34449)
0812 07:04:25.077179 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:25.077187 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:25.077783 (+ 596us) server_negotiation.cc:407] Connection header received
0812 07:04:25.078037 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:25.078045 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:25.078180 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:25.078312 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:25.079935 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.080816 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.083474 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.084103 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.084237 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:25.088219 (+ 3982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:25.088227 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:25.088233 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:25.088301 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:25.091572 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:25.092253 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:25.092257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:25.092260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:25.092372 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:25.092709 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:25.092712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:25.092714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:25.093016 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:25.093125 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:25.093556 (+ 431us) server_negotiation.cc:299] Negotiation successful
0812 07:04:25.093751 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":292,"threads_started":1}
I20260812 07:04:25.094062 14261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:25.076571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.0.0.1:60032)
0812 07:04:25.077627 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:25.077670 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:25.077863 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:25.078392 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:25.078398 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:25.079312 (+ 914us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:25.079754 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:25.079767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.081005 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.081010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:25.082972 (+ 1962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:25.082990 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.084271 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.084276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:25.084434 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:25.085037 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:25.085063 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:25.087910 (+ 2847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:25.091691 (+ 3781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:25.091702 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:25.091708 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:25.092142 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:25.092464 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:25.092467 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:25.092470 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:25.092599 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:25.093124 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:25.093129 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:25.093274 (+ 145us) client_negotiation.cc:769] Sending connection context
0812 07:04:25.093454 (+ 180us) client_negotiation.cc:240] Negotiation successful
0812 07:04:25.093687 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"thread_start_us":612,"threads_started":1}
I20260812 07:04:25.095649 14252 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 07:04:25 dist-test-slave-69rg krb5kdc[13585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518264, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.61@KRBTEST.COM for oryx/127.6.61.61@KRBTEST.COM
I20260812 07:04:25.108215 14261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:25.096395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:39641 (local address 127.0.0.1:60730)
0812 07:04:25.096565 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:25.096581 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:25.096698 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:25.096985 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:25.096989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:25.097343 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:25.097477 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:25.097486 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.098542 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.098550 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:25.099420 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:25.099431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.100154 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.100159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:25.100356 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:25.100919 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:25.100936 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:25.103123 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:25.105743 (+ 2620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:25.105751 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:25.105755 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:25.106183 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:25.106555 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:25.106559 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:25.106564 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:25.106701 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:25.107231 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:25.107235 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:25.107398 (+ 163us) client_negotiation.cc:769] Sending connection context
0812 07:04:25.107536 (+ 138us) client_negotiation.cc:240] Negotiation successful
0812 07:04:25.107840 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260812 07:04:25.108316 14258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:25.096590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60730 (local address 127.6.61.61:39641)
0812 07:04:25.096735 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:25.096743 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:25.096760 (+ 17us) server_negotiation.cc:407] Connection header received
0812 07:04:25.096808 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:25.096812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:25.096867 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:25.096967 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:25.097614 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.098416 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.099589 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.100002 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.100127 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:25.103288 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:25.103302 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:25.103305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:25.103337 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:25.105613 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:25.106306 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:25.106311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:25.106315 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:25.106442 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:25.106809 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:25.106816 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:25.106819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:25.107088 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:25.107267 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:25.107628 (+ 361us) server_negotiation.cc:299] Negotiation successful
0812 07:04:25.107995 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":151,"server-negotiator.queue_time_us":49}
I20260812 07:04:25.113396 14252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa: Bootstrap starting.
I20260812 07:04:25.116889 14252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:25.117990 14252 log.cc:824] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:25.120847 14252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa: No bootstrap required, opened a new log
I20260812 07:04:25.128003 14252 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } }
I20260812 07:04:25.128369 14252 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:25.128489 14252 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06a305d836de4590a62f589f803f0dfa, State: Initialized, Role: FOLLOWER
I20260812 07:04:25.129169 14252 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } }
I20260812 07:04:25.130115 14263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } } }
I20260812 07:04:25.130457 14263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:04:25.131188 14252 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [sys.catalog]: configured and running, proceeding with master startup.
W20260812 07:04:25.141774 14274 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 07:04:25.142043 14274 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 07:04:25.144275 14179 master_runner.cc:428] Detected that this master 06a305d836de4590a62f589f803f0dfa is joining an existing cluster
I20260812 07:04:25.144397 14179 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.61.61:39641
I20260812 07:04:25.150097 14263 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:04:25.150408 14263 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } }
I20260812 07:04:25.152190 14263 leader_election.cc:288] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c86051a8e4146a3b059183c03351248 (127.6.61.62:34449)
I20260812 07:04:25.153060 13651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "06a305d836de4590a62f589f803f0dfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c86051a8e4146a3b059183c03351248" is_pre_election: true
I20260812 07:04:25.154743 14196 leader_election.cc:302] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 06a305d836de4590a62f589f803f0dfa; no voters: 6c86051a8e4146a3b059183c03351248
I20260812 07:04:25.155742 14263 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 07:04:25.179787 14258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:25.149407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60738 (local address 127.6.61.61:39641)
0812 07:04:25.149656 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:25.149666 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:25.162411 (+ 12745us) server_negotiation.cc:407] Connection header received
0812 07:04:25.162622 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:25.162628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:25.162698 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:25.162785 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:25.164538 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.165330 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.167395 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.168197 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.168338 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:25.171533 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:25.171541 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:25.171546 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:25.171590 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:25.175611 (+ 4021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:25.176655 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:25.176666 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:25.176670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:25.176815 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:25.177235 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:25.177241 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:25.177244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:25.177996 (+ 752us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:25.178178 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:25.178677 (+ 499us) server_negotiation.cc:299] Negotiation successful
0812 07:04:25.179205 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":77}
I20260812 07:04:25.181174 14282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:25.149516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:39641 (local address 127.0.0.1:60738)
0812 07:04:25.162313 (+ 12797us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:25.162332 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:25.162472 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:25.162973 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:25.162978 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:25.163862 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:25.164369 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:25.164385 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.165517 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.165526 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:25.167221 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:25.167239 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.168642 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.168648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:25.168933 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:25.169685 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:25.169708 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:25.171024 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:25.175774 (+ 4750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:25.175782 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:25.175787 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:25.176522 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:25.176949 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:25.176954 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:25.176958 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:25.177123 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:25.178246 (+ 1123us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:25.178250 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:25.178401 (+ 151us) client_negotiation.cc:769] Sending connection context
0812 07:04:25.180372 (+ 1971us) client_negotiation.cc:240] Negotiation successful
0812 07:04:25.180707 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12666,"mutex_wait_us":106,"thread_start_us":12477,"threads_started":1}
WARNING: no policy specified for oryx/127.6.61.60@KRBTEST.COM; defaulting to no policy
I20260812 07:04:25.184573 14281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:25.149434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.0.0.1:60044)
0812 07:04:25.161881 (+ 12447us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:25.161911 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:25.162453 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:25.163116 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:25.163123 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:25.164022 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:25.164380 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:25.164390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.166785 (+ 2395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.166794 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:25.168913 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:25.168934 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.170466 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.170475 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:25.170798 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:25.171542 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:25.171573 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:25.176516 (+ 4943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:25.181211 (+ 4695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:25.181217 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:25.181221 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:25.181760 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:25.182355 (+ 595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:25.182362 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:25.182368 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:25.182556 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:25.183424 (+ 868us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:25.183428 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:25.183645 (+ 217us) client_negotiation.cc:769] Sending connection context
0812 07:04:25.183819 (+ 174us) client_negotiation.cc:240] Negotiation successful
0812 07:04:25.184181 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12213,"mutex_wait_us":72,"spinlock_wait_cycles":31104,"thread_start_us":12637,"threads_started":1}
I20260812 07:04:25.184633 14259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:25.149956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60044 (local address 127.6.61.62:34449)
0812 07:04:25.150150 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:25.150158 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:25.162047 (+ 11889us) server_negotiation.cc:407] Connection header received
0812 07:04:25.162646 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:25.162654 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:25.162759 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:25.162892 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:25.165495 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.166578 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.169114 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.169945 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.170064 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:25.176793 (+ 6729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:25.176803 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:25.176811 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:25.176863 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:25.180893 (+ 4030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:25.181965 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:25.181974 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:25.181980 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:25.182168 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:25.182701 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:25.182706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:25.182709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:25.183143 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:25.183294 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:25.183988 (+ 694us) server_negotiation.cc:299] Negotiation successful
0812 07:04:25.184249 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.61.60@KRBTEST.COM" created.
Aug 12 07:04:25 dist-test-slave-69rg krb5kdc[13585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518262, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.62@KRBTEST.COM for oryx/127.6.61.61@KRBTEST.COM
I20260812 07:04:25.213735 14258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:25.191887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60740 (local address 127.6.61.61:39641)
0812 07:04:25.192083 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:25.192091 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:25.195204 (+ 3113us) server_negotiation.cc:407] Connection header received
0812 07:04:25.195644 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:25.195652 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:25.195796 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:25.195921 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:25.197521 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.198364 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.200306 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.201006 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.201140 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:25.205762 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:25.205772 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:25.205780 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:25.205898 (+ 118us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:25.210139 (+ 4241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:25.211070 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:25.211078 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:25.211085 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:25.211279 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:25.211769 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:25.211777 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:25.211783 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:25.212201 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:25.212386 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:25.212941 (+ 555us) server_negotiation.cc:299] Negotiation successful
0812 07:04:25.213275 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 07:04:25.213765 14284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:25.191831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:39641 (local address 127.0.0.1:60740)
0812 07:04:25.195065 (+ 3234us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:25.195111 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:25.195383 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:25.195992 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:25.196002 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:25.196948 (+ 946us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:25.197283 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:25.197305 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.198512 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.198518 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:25.200126 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:25.200147 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.201206 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.201214 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:25.201489 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:25.202318 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:25.202355 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:25.205540 (+ 3185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:25.210302 (+ 4762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:25.210316 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:25.210325 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:25.210918 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:25.211434 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:25.211441 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:25.211445 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:25.211632 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:25.212359 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:25.212368 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:25.212586 (+ 218us) client_negotiation.cc:769] Sending connection context
0812 07:04:25.212805 (+ 219us) client_negotiation.cc:240] Negotiation successful
0812 07:04:25.213226 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2981,"thread_start_us":2836,"threads_started":1}
I20260812 07:04:25.216338 13639 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.61.61:39641: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "06a305d836de4590a62f589f803f0dfa"
member_type: NON_VOTER
last_known_addr {
host: "127.6.61.61"
port: 39641
}
attrs {
promote: true
}
}
dest_uuid: "6c86051a8e4146a3b059183c03351248"
cas_config_opid_index: -1
I20260812 07:04:25.217473 13639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: NON_VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: true } }
I20260812 07:04:25.221127 14288 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 06a305d836de4590a62f589f803f0dfa (127.6.61.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: NON_VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: true } } }
I20260812 07:04:25.222309 14223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.61.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.60.keytab.
Entry for principal oryx/127.6.61.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.60.keytab.
I20260812 07:04:25.225983 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.60:34607
--webserver_interface=127.6.61.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.60.keytab
--principal=oryx/127.6.61.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:34261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:34449,127.6.61.61:39641,127.6.61.60:34607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:04:25.226801 14223 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 1 FOLLOWER]: Refusing update from remote peer 6c86051a8e4146a3b059183c03351248: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 07:04:25.227751 14287 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: NON_VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:04:25.235164 14288 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.61.61:39641
I20260812 07:04:25.236886 14288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6c86051a8e4146a3b059183c03351248" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: NON_VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: true } } }
I20260812 07:04:25.237174 14288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:25.237452 14179 master.cc:537] Master@127.6.61.61:39641 shutting down...
I20260812 07:04:25.262171 14223 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 06a305d836de4590a62f589f803f0dfa (127.6.61.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: NON_VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: true } } }
I20260812 07:04:25.273943 14298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c86051a8e4146a3b059183c03351248" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: NON_VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: true } } }
I20260812 07:04:25.274505 14298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [sys.catalog]: This master's current role is: LEARNER
I20260812 07:04:25.274746 14294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c86051a8e4146a3b059183c03351248" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: NON_VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: true } } }
I20260812 07:04:25.275111 14294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [sys.catalog]: This master's current role is: LEARNER
I20260812 07:04:25.274513 14263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c86051a8e4146a3b059183c03351248. Latest consensus state: current_term: 1 leader_uuid: "6c86051a8e4146a3b059183c03351248" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: NON_VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: true } } }
I20260812 07:04:25.275779 14263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [sys.catalog]: This master's current role is: LEARNER
I20260812 07:04:25.277416 14179 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 1 LEARNER]: Raft consensus shutting down.
I20260812 07:04:25.277938 14179 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 1 LEARNER]: Raft consensus is shut down!
I20260812 07:04:25.278124 14179 tablet_replica.cc:330] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa: stopping tablet replica
I20260812 07:04:25.285569 14179 master.cc:552] Master@127.6.61.61:39641 shutdown complete.
I20260812 07:04:25.285737 14179 master_runner.cc:305] Clearing existing system tablet
I20260812 07:04:25.286195 14179 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:25.286316 14179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:25.288712 14179 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260812 07:04:25.290230 14300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:25.290552 14179 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:25.290851 14179 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 07:04:25.290941 14179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-1/wal
uuid: "06a305d836de4590a62f589f803f0dfa"
format_stamp: "Formatted at 2026-08-12 07:04:24 on dist-test-slave-69rg"
server_key: "8a6c7fbeeefdecc14c21c3d9cb9342c7"
server_key_iv: "597105f727623042f2cb517fee2b6e78"
server_key_version: "encryptionkey@0"
I20260812 07:04:25.293990 14179 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:04:25.298913 14179 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 07:04:25.299105 14179 log.cc:1196] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260812 07:04:25.299531 14179 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa: Deleting consensus metadata
I20260812 07:04:25.299806 14179 master_runner.cc:315] Copying system tablet from 127.6.61.62:34449
I20260812 07:04:25.302552 14179 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa: tablet copy: Beginning tablet copy session from remote peer at address 127.6.61.62:34449
I20260812 07:04:25.316102 14259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:25.303438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60054 (local address 127.6.61.62:34449)
0812 07:04:25.303669 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:25.303679 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:25.304211 (+ 532us) server_negotiation.cc:407] Connection header received
0812 07:04:25.304439 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:25.304445 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:25.304541 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:25.304657 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:25.305509 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.306376 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.307672 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.308113 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.308262 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:25.310316 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:25.310321 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:25.310325 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:25.310360 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:25.313183 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:25.314028 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:25.314035 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:25.314039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:25.314174 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:25.314583 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:25.314588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:25.314593 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:25.314884 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:25.315052 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:25.315474 (+ 422us) server_negotiation.cc:299] Negotiation successful
0812 07:04:25.315690 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 07:04:25.316138 14307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:25.303392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.0.0.1:60054)
0812 07:04:25.304084 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:25.304108 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:25.304288 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:25.304690 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:25.304695 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:25.305116 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:25.305345 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:25.305356 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.306554 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.306560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:25.307517 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:25.307530 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.308287 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.308293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:25.308500 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:25.309195 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:25.309236 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:25.310106 (+ 870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:25.313324 (+ 3218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:25.313330 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:25.313334 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:25.313904 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:25.314303 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:25.314309 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:25.314312 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:25.314467 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:25.315014 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:25.315019 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:25.315191 (+ 172us) client_negotiation.cc:769] Sending connection context
0812 07:04:25.315346 (+ 155us) client_negotiation.cc:240] Negotiation successful
0812 07:04:25.315690 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":421,"threads_started":1}
I20260812 07:04:25.317206 13664 tablet_copy_service.cc:140] P 6c86051a8e4146a3b059183c03351248: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 06a305d836de4590a62f589f803f0dfa ({username='oryx', principal='oryx/127.6.61.61@KRBTEST.COM'} at 127.0.0.1:60054)
I20260812 07:04:25.317457 13664 tablet_copy_service.cc:161] P 6c86051a8e4146a3b059183c03351248: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 06a305d836de4590a62f589f803f0dfa at {username='oryx', principal='oryx/127.6.61.61@KRBTEST.COM'} at 127.0.0.1:60054: session id = 06a305d836de4590a62f589f803f0dfa-00000000000000000000000000000000
I20260812 07:04:25.321128 13664 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 07:04:25.323019 14179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 07:04:25.327639 14179 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa: tablet copy: Starting download of 0 data blocks...
I20260812 07:04:25.327943 14179 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa: tablet copy: Starting download of 1 WAL segments...
I20260812 07:04:25.330312 14179 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 07:04:25.334244 13664 tablet_copy_service.cc:342] P 6c86051a8e4146a3b059183c03351248: Request end of tablet copy session 06a305d836de4590a62f589f803f0dfa-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.61.61@KRBTEST.COM'} at 127.0.0.1:60054
I20260812 07:04:25.334483 13664 tablet_copy_service.cc:433] P 6c86051a8e4146a3b059183c03351248: ending tablet copy session 06a305d836de4590a62f589f803f0dfa-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 06a305d836de4590a62f589f803f0dfa
I20260812 07:04:25.345119 14179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:25.347914 14310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:25.354048 14311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:25.362285 14179 server_base.cc:1034] running on GCE node
W20260812 07:04:25.362936 14313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:25.363451 14179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:25.364111 14179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:25.365377 14179 hybrid_clock.cc:648] HybridClock initialized: now 1786518265365316 us; error 38 us; skew 500 ppm
I20260812 07:04:25.365613 14179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:04:25 dist-test-slave-69rg krb5kdc[13585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518265, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:25.367633 14179 init.cc:377] Logged in from keytab as oryx/127.6.61.61@KRBTEST.COM (short username oryx)
I20260812 07:04:25.368980 14179 webserver.cc:460] Webserver started at http://127.6.61.61:46329/ using document root <none> and password file <none>
I20260812 07:04:25.369328 14179 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:25.369431 14179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:25.372167 14179 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260812 07:04:25.373910 14328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:25.374390 14179 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 07:04:25.374549 14179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-1/wal
uuid: "06a305d836de4590a62f589f803f0dfa"
format_stamp: "Formatted at 2026-08-12 07:04:24 on dist-test-slave-69rg"
server_key: "8a6c7fbeeefdecc14c21c3d9cb9342c7"
server_key_iv: "597105f727623042f2cb517fee2b6e78"
server_key_version: "encryptionkey@0"
I20260812 07:04:25.374718 14179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:25.393105 14179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:25.396747 14179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:25.440042 14179 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.61:39641
I20260812 07:04:25.440088 14384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.61:39641 every 8 connection(s)
I20260812 07:04:25.441041 14179 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-1/data/info.pb
I20260812 07:04:25.448619 14386 sys_catalog.cc:263] Verifying existing consensus state
I20260812 07:04:25.451208 14386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa: Bootstrap starting.
I20260812 07:04:25.461613 14386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 07:04:25.462077 14386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa: Bootstrap complete.
I20260812 07:04:25.462996 14386 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: NON_VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: true } }
I20260812 07:04:25.463233 14386 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 06a305d836de4590a62f589f803f0dfa, State: Initialized, Role: LEARNER
I20260812 07:04:25.463521 14386 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: NON_VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: true } }
I20260812 07:04:25.464324 14391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: NON_VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: true } } }
I20260812 07:04:25.464668 14386 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:04:25.464751 14391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [sys.catalog]: This master's current role is: LEARNER
I20260812 07:04:25.473165 14403 catalog_manager.cc:1259] Loaded cluster ID: 71d35a330bec42eb92f085ce763b979e
I20260812 07:04:25.473289 14403 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa: loading cluster ID for follower catalog manager: success
I20260812 07:04:25.476114 14403 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa: acquiring CA information for follower catalog manager: success
I20260812 07:04:25.478170 14403 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 07:04:25.538551 14171 heartbeater.cc:503] Master 127.6.61.62:34449 was elected leader, sending a full tablet report...
I20260812 07:04:25.658128 14288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c86051a8e4146a3b059183c03351248" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: NON_VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: true } } }
I20260812 07:04:25.658455 14288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:25.666152 14284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:25.659180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:39641 (local address 127.0.0.1:60754)
0812 07:04:25.659368 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:25.659392 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:25.659547 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:25.660621 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:25.660628 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:25.660661 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:25.660867 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:25.660878 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.662408 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.662414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:25.663698 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:25.663713 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.665166 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.665173 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:25.665500 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:25.665550 (+ 50us) client_negotiation.cc:769] Sending connection context
0812 07:04:25.665667 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 07:04:25.665867 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260812 07:04:25.666402 14405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:25.659396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60754 (local address 127.6.61.61:39641)
0812 07:04:25.660255 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:25.660259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:25.660281 (+ 22us) server_negotiation.cc:407] Connection header received
0812 07:04:25.660347 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:25.660351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:25.660421 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:25.660579 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:25.661044 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.662253 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.663870 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.664948 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.665109 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:25.665379 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:25.665843 (+ 464us) server_negotiation.cc:299] Negotiation successful
0812 07:04:25.665987 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":528,"threads_started":1}
I20260812 07:04:25.667663 14391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c86051a8e4146a3b059183c03351248. Latest consensus state: current_term: 1 leader_uuid: "6c86051a8e4146a3b059183c03351248" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: NON_VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: true } } }
I20260812 07:04:25.667984 14391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [sys.catalog]: This master's current role is: LEARNER
I20260812 07:04:25.668672 14287 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248: attempting to promote NON_VOTER 06a305d836de4590a62f589f803f0dfa to VOTER
I20260812 07:04:25.669622 14287 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } }
I20260812 07:04:25.671622 14360 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 1 LEARNER]: Refusing update from remote peer 6c86051a8e4146a3b059183c03351248: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 07:04:25.672264 14404 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 07:04:25.675483 14287 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 06a305d836de4590a62f589f803f0dfa (127.6.61.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } }
I20260812 07:04:25.678350 14287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6c86051a8e4146a3b059183c03351248" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } }
I20260812 07:04:25.678570 14287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:25.678893 14404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c86051a8e4146a3b059183c03351248" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } }
I20260812 07:04:25.679149 14404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:25.679478 14360 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 06a305d836de4590a62f589f803f0dfa (127.6.61.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } }
I20260812 07:04:25.682638 14391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c86051a8e4146a3b059183c03351248" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } }
I20260812 07:04:25.682885 14391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [sys.catalog]: This master's current role is: FOLLOWER
W20260812 07:04:25.696295 14291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:25.696698 14291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:25.696846 14291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:25.705214 14291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:04:25.705335 14291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:25.705384 14291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:25.705443 14291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:04:25.705487 14291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:04:25.715894 14291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:34261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:34449,127.6.61.61:39641,127.6.61.60:34607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.60.keytab
--principal=oryx/127.6.61.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.60:34607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.61.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:25.717340 14291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:25.718796 14291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:25.730890 14412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:25.730821 14416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:25.730924 14413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:25.731741 14291 server_base.cc:1034] running on GCE node
I20260812 07:04:25.732501 14291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:25.733906 14291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:25.735118 14291 hybrid_clock.cc:648] HybridClock initialized: now 1786518265735045 us; error 66 us; skew 500 ppm
I20260812 07:04:25.735417 14291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:04:25 dist-test-slave-69rg krb5kdc[13585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518265, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:25.738560 14291 init.cc:377] Logged in from keytab as oryx/127.6.61.60@KRBTEST.COM (short username oryx)
I20260812 07:04:25.740077 14291 webserver.cc:460] Webserver started at http://127.6.61.60:41075/ using document root <none> and password file <none>
I20260812 07:04:25.740759 14291 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:25.740908 14291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:25.741215 14291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:25.744340 14291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-2/data/instance:
uuid: "b9758e0d5bf44fa68a4085ab311c9f43"
format_stamp: "Formatted at 2026-08-12 07:04:25 on dist-test-slave-69rg"
server_key: "3eea9695e87c34a4b43e82cc982ef5f9"
server_key_iv: "34f74048281814c0c318440c459e355b"
server_key_version: "encryptionkey@0"
I20260812 07:04:25.745136 14291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-2/wal/instance:
uuid: "b9758e0d5bf44fa68a4085ab311c9f43"
format_stamp: "Formatted at 2026-08-12 07:04:25 on dist-test-slave-69rg"
server_key: "3eea9695e87c34a4b43e82cc982ef5f9"
server_key_iv: "34f74048281814c0c318440c459e355b"
server_key_version: "encryptionkey@0"
I20260812 07:04:25.750335 14291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 07:04:25.753758 14423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:25.755107 14291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:04:25.755322 14291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-2/wal
uuid: "b9758e0d5bf44fa68a4085ab311c9f43"
format_stamp: "Formatted at 2026-08-12 07:04:25 on dist-test-slave-69rg"
server_key: "3eea9695e87c34a4b43e82cc982ef5f9"
server_key_iv: "34f74048281814c0c318440c459e355b"
server_key_version: "encryptionkey@0"
I20260812 07:04:25.755517 14291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:25.785497 14291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:25.789788 14291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:25.790128 14291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:25.815147 14291 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.60:34607
I20260812 07:04:25.815177 14482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.60:34607 every 8 connection(s)
I20260812 07:04:25.816884 14291 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-2/data/info.pb
I20260812 07:04:25.818729 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 14291
I20260812 07:04:25.822705 14483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:25.829738 14483 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 07:04:25 dist-test-slave-69rg krb5kdc[13585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.61.60@KRBTEST.COM
Aug 12 07:04:25 dist-test-slave-69rg krb5kdc[13585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518265, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.60@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM
I20260812 07:04:25.845134 14486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:25.821264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44648 (local address 127.6.61.60:34607)
0812 07:04:25.822291 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:25.822306 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:25.822356 (+ 50us) server_negotiation.cc:407] Connection header received
0812 07:04:25.824256 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:25.824281 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:25.824660 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:25.825257 (+ 597us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:25.826935 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.828324 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.829629 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.830429 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.830621 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:25.836421 (+ 5800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:25.836429 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:25.836441 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:25.836477 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:25.840561 (+ 4084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:25.841300 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:25.841307 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:25.841317 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:25.841502 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:25.841985 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:25.841991 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:25.842000 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:25.842718 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:25.843020 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:25.843615 (+ 595us) server_negotiation.cc:299] Negotiation successful
0812 07:04:25.844071 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":785,"thread_start_us":501,"threads_started":1}
I20260812 07:04:25.850360 14490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:25.832739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60062 (local address 127.6.61.62:34449)
0812 07:04:25.834237 (+ 1498us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:25.834241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:25.834257 (+ 16us) server_negotiation.cc:407] Connection header received
0812 07:04:25.834309 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:25.834312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:25.834370 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:25.834554 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:25.835883 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.836866 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.839309 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.840009 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.840169 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:25.844045 (+ 3876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:25.844050 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:25.844054 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:25.844095 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:25.847456 (+ 3361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:25.848464 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:25.848469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:25.848471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:25.848616 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:25.849026 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:25.849029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:25.849031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:25.849298 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:25.849410 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:25.849921 (+ 511us) server_negotiation.cc:299] Negotiation successful
0812 07:04:25.850099 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1386,"thread_start_us":1260,"threads_started":1}
I20260812 07:04:25.850359 14487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:25.832547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.0.0.1:60062)
0812 07:04:25.833554 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:25.833606 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:25.833988 (+ 382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:25.834686 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:25.834697 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:25.835416 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:25.835705 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:25.835715 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.837046 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.837055 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:25.839100 (+ 2045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:25.839114 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.840243 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.840248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:25.840406 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:25.841075 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:25.841096 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:25.843825 (+ 2729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:25.847594 (+ 3769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:25.847603 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:25.847608 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:25.848296 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:25.848762 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:25.848765 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:25.848768 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:25.848912 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:25.849421 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:25.849426 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:25.849567 (+ 141us) client_negotiation.cc:769] Sending connection context
0812 07:04:25.849727 (+ 160us) client_negotiation.cc:240] Negotiation successful
0812 07:04:25.850068 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":450,"threads_started":1}
I20260812 07:04:25.851882 14483 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 07:04:25 dist-test-slave-69rg krb5kdc[13585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518265, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.60@KRBTEST.COM for oryx/127.6.61.61@KRBTEST.COM
I20260812 07:04:25.865092 14487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:25.852575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:39641 (local address 127.0.0.1:60764)
0812 07:04:25.852814 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:25.852833 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:25.852931 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:25.853227 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:25.853231 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:25.853531 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:25.853686 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:25.853694 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.854874 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.854878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:25.855822 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:25.855843 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.856700 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.856705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:25.856853 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:25.857444 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:25.857462 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:25.859583 (+ 2121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:25.862578 (+ 2995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:25.862582 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:25.862584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:25.863027 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:25.863541 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:25.863544 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:25.863547 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:25.863700 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:25.864293 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:25.864297 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:25.864435 (+ 138us) client_negotiation.cc:769] Sending connection context
0812 07:04:25.864549 (+ 114us) client_negotiation.cc:240] Negotiation successful
0812 07:04:25.864776 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260812 07:04:25.865175 14405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:25.852709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60764 (local address 127.6.61.61:39641)
0812 07:04:25.852897 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:25.852901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:25.852918 (+ 17us) server_negotiation.cc:407] Connection header received
0812 07:04:25.853036 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:25.853040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:25.853101 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:25.853183 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:25.853915 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.854732 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.856002 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.856528 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.856651 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:25.859792 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:25.859797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:25.859800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:25.859835 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:25.862432 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:25.863205 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:25.863210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:25.863213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:25.863385 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:25.863819 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:25.863824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:25.863826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:25.864176 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:25.864308 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:25.864662 (+ 354us) server_negotiation.cc:299] Negotiation successful
0812 07:04:25.864934 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 07:04:25.866677 14483 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.61.60" port: 34607 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 07:04:25 dist-test-slave-69rg krb5kdc[13585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518265, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.60@KRBTEST.COM for oryx/127.6.61.60@KRBTEST.COM
I20260812 07:04:25.879401 14487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:25.867506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:34607 (local address 127.0.0.1:44662)
0812 07:04:25.867784 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:25.867805 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:25.867914 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:25.868196 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:25.868201 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:25.868576 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:25.868710 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:25.868718 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.869684 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.869691 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:25.870698 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:25.870710 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.871403 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.871410 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:25.871676 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:25.872197 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:25.872213 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:25.874256 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:25.877303 (+ 3047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:25.877307 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:25.877310 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:25.877658 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:25.878043 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:25.878046 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:25.878049 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:25.878212 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:25.878681 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:25.878684 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:25.878803 (+ 119us) client_negotiation.cc:769] Sending connection context
0812 07:04:25.878909 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 07:04:25.879099 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260812 07:04:25.879582 14486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:25.867731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44662 (local address 127.6.61.60:34607)
0812 07:04:25.867885 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:25.867891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:25.867919 (+ 28us) server_negotiation.cc:407] Connection header received
0812 07:04:25.868009 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:25.868015 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:25.868078 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:25.868175 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:25.868858 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.869558 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.870833 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.871229 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.871366 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:25.874460 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:25.874465 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:25.874469 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:25.874531 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:25.877174 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:25.877759 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:25.877763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:25.877765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:25.877943 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:25.878313 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:25.878317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:25.878319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:25.878580 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:25.878689 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:25.879004 (+ 315us) server_negotiation.cc:299] Negotiation successful
0812 07:04:25.879222 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":31}
I20260812 07:04:25.884727 14483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43: Bootstrap starting.
I20260812 07:04:25.888167 14483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:25.889293 14483 log.cc:824] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:25.892387 14483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43: No bootstrap required, opened a new log
I20260812 07:04:25.900110 14483 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 34607 } }
I20260812 07:04:25.900470 14483 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:25.900604 14483 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9758e0d5bf44fa68a4085ab311c9f43, State: Initialized, Role: FOLLOWER
I20260812 07:04:25.901147 14483 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 34607 } }
I20260812 07:04:25.902164 14493 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 34607 } } }
I20260812 07:04:25.902552 14493 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:04:25.903290 14483 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 07:04:25.913147 14505 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 07:04:25.913307 14505 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 07:04:25.917891 14291 master_runner.cc:428] Detected that this master b9758e0d5bf44fa68a4085ab311c9f43 is joining an existing cluster
I20260812 07:04:25.918068 14291 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.61.60:34607
I20260812 07:04:25.948410 14490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:25.923488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60072 (local address 127.6.61.62:34449)
0812 07:04:25.923675 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:25.923682 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:25.925755 (+ 2073us) server_negotiation.cc:407] Connection header received
0812 07:04:25.926104 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:25.926110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:25.926217 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:25.926296 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:25.928179 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.929656 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.932653 (+ 2997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.933559 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.933677 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:25.940654 (+ 6977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:25.940661 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:25.940665 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:25.940707 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:25.944930 (+ 4223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:25.945746 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:25.945751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:25.945755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:25.945946 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:25.946480 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:25.946488 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:25.946491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:25.946941 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:25.947082 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:25.947729 (+ 647us) server_negotiation.cc:299] Negotiation successful
0812 07:04:25.948005 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 07:04:25.949334 14511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:25.923435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.0.0.1:60072)
0812 07:04:25.925421 (+ 1986us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:25.925445 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:25.925937 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:25.926580 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:25.926588 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:25.927474 (+ 886us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:25.927956 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:25.927973 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.929946 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.929955 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:25.932441 (+ 2486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:25.932462 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.934115 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.934125 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:25.934381 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:25.935298 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:25.935325 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:25.940419 (+ 5094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:25.945098 (+ 4679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:25.945104 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:25.945109 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:25.945614 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:25.946108 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:25.946114 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:25.946119 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:25.946332 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:25.947182 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:25.947188 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:25.947424 (+ 236us) client_negotiation.cc:769] Sending connection context
0812 07:04:25.948411 (+ 987us) client_negotiation.cc:240] Negotiation successful
0812 07:04:25.948772 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"thread_start_us":662,"threads_started":1}
I20260812 07:04:25.952270 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 13709
I20260812 07:04:25.952575 14405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:25.923337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60780 (local address 127.6.61.61:39641)
0812 07:04:25.923543 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:25.923550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:25.935344 (+ 11794us) server_negotiation.cc:407] Connection header received
0812 07:04:25.935760 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:25.935766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:25.935859 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:25.935932 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:25.937642 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.938792 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.940212 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.940896 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.941035 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:25.944214 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:25.944221 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:25.944225 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:25.944272 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:25.947844 (+ 3572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:25.950116 (+ 2272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:25.950121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:25.950124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:25.950298 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:25.950699 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:25.950707 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:25.950711 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:25.951088 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:25.951247 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:25.951907 (+ 660us) server_negotiation.cc:299] Negotiation successful
0812 07:04:25.952156 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 07:04:25.959405 14486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:25.925031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44664 (local address 127.6.61.60:34607)
0812 07:04:25.925207 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:25.925213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:25.937141 (+ 11928us) server_negotiation.cc:407] Connection header received
0812 07:04:25.937301 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:25.937307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:25.937386 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:25.937498 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:25.938779 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.939968 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.941972 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.942747 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.942865 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:25.945731 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:25.945737 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:25.945741 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:25.945785 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:25.954557 (+ 8772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:25.955625 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:25.955633 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:25.955639 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:25.955826 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:25.956259 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:25.956263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:25.956267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:25.957072 (+ 805us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:25.957292 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:25.958381 (+ 1089us) server_negotiation.cc:299] Negotiation successful
0812 07:04:25.958671 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 07:04:25.960695 14513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:25.923571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:39641 (local address 127.0.0.1:60780)
0812 07:04:25.935084 (+ 11513us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:25.935108 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:25.935583 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:25.936153 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:25.936159 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:25.937035 (+ 876us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:25.937434 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:25.937445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.938947 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.938951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:25.940020 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:25.940059 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.941295 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.941302 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:25.941474 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:25.942345 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:25.942374 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:25.943982 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:25.948024 (+ 4042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:25.948029 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:25.948033 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:25.949973 (+ 1940us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:25.950436 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:25.950439 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:25.950441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:25.950571 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:25.951329 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:25.951334 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:25.951537 (+ 203us) client_negotiation.cc:769] Sending connection context
0812 07:04:25.952838 (+ 1301us) client_negotiation.cc:240] Negotiation successful
0812 07:04:25.953192 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11348,"thread_start_us":12995,"threads_started":1}
I20260812 07:04:25.974663 14512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:25.924598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:34607 (local address 127.0.0.1:44664)
0812 07:04:25.936875 (+ 12277us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:25.936895 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:25.937166 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:25.937539 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:25.937543 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:25.938281 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:25.938614 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:25.938625 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.940149 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.940160 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:25.941797 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:25.941837 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.943121 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.943127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:25.943334 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:25.944047 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:25.944088 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:25.945543 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:25.954721 (+ 9178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:25.954729 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:25.954735 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:25.955470 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:25.955965 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:25.955972 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:25.955978 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:25.956135 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:25.957355 (+ 1220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:25.957363 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:25.957598 (+ 235us) client_negotiation.cc:769] Sending connection context
0812 07:04:25.957778 (+ 180us) client_negotiation.cc:240] Negotiation successful
0812 07:04:25.960456 (+ 2678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12126,"thread_start_us":10766,"threads_started":1}
Aug 12 07:04:25 dist-test-slave-69rg krb5kdc[13585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518262, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.62@KRBTEST.COM for oryx/127.6.61.60@KRBTEST.COM
Aug 12 07:04:25 dist-test-slave-69rg krb5kdc[13585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.61.1@KRBTEST.COM
I20260812 07:04:25.994760 14284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:25.956652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:34607 (local address 127.0.0.1:44672)
0812 07:04:25.957173 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:25.957192 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:25.957312 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:25.968081 (+ 10769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:25.968087 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:25.968861 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:25.969356 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:25.969375 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.972061 (+ 2686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.972071 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:25.974648 (+ 2577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:25.974669 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.978843 (+ 4174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.978854 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:25.979133 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:25.980039 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:25.980080 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:25.984350 (+ 4270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:25.991026 (+ 6676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:25.991031 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:25.991035 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:25.991649 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:25.992284 (+ 635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:25.992291 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:25.992296 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:25.992512 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:25.993330 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:25.993335 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:25.993570 (+ 235us) client_negotiation.cc:769] Sending connection context
0812 07:04:25.993765 (+ 195us) client_negotiation.cc:240] Negotiation successful
0812 07:04:25.994167 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393}
I20260812 07:04:25.995627 14514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:25.953741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58164 (local address 127.6.61.1:42623)
0812 07:04:25.967056 (+ 13315us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:25.967070 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:25.967126 (+ 56us) server_negotiation.cc:407] Connection header received
0812 07:04:25.967212 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:25.967222 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:25.967558 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:25.971105 (+ 3547us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:25.971400 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.972843 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.974548 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.975952 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.984817 (+ 8865us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:25.986928 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:25.986943 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:25.986960 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:25.987011 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:25.991079 (+ 4068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:25.992075 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:25.992083 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:25.992087 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:25.992390 (+ 303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:25.992939 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:25.992944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:25.992947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:25.993467 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:25.993972 (+ 505us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:25.994383 (+ 411us) server_negotiation.cc:299] Negotiation successful
0812 07:04:25.994787 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12751,"thread_start_us":13010,"threads_started":1}
I20260812 07:04:25.996145 14486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:25.957522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44672 (local address 127.6.61.60:34607)
0812 07:04:25.960003 (+ 2481us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:25.960011 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:25.960034 (+ 23us) server_negotiation.cc:407] Connection header received
0812 07:04:25.960147 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:25.960155 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:25.960602 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:25.969570 (+ 8968us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:25.969882 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.971519 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.974869 (+ 3350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:25.978263 (+ 3394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:25.978476 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:25.984636 (+ 6160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:25.984644 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:25.984649 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:25.984697 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:25.990828 (+ 6131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:25.991845 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:25.991851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:25.991854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:25.992093 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:25.992639 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:25.992649 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:25.992652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:25.993143 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:25.993296 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:25.995294 (+ 1998us) server_negotiation.cc:299] Negotiation successful
0812 07:04:25.995609 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2324,"thread_start_us":22546,"threads_started":1}
I20260812 07:04:25.997431 13852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:25.998667 13639 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.61.60:34607: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43"
member_type: NON_VOTER
last_known_addr {
host: "127.6.61.60"
port: 34607
}
attrs {
promote: true
}
}
dest_uuid: "6c86051a8e4146a3b059183c03351248"
cas_config_opid_index: 6
I20260812 07:04:25.999928 13639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: NON_VOTER last_known_addr { host: "127.6.61.60" port: 34607 } attrs { promote: true } }
I20260812 07:04:26.005672 14454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:26.005699 14360 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 1 FOLLOWER]: Refusing update from remote peer 6c86051a8e4146a3b059183c03351248: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260812 07:04:26.006678 14287 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260812 07:04:26.010403 14287 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b9758e0d5bf44fa68a4085ab311c9f43 (127.6.61.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: NON_VOTER last_known_addr { host: "127.6.61.60" port: 34607 } attrs { promote: true } } }
I20260812 07:04:26.011837 14454 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [term 1 FOLLOWER]: Refusing update from remote peer 6c86051a8e4146a3b059183c03351248: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260812 07:04:26.013239 14288 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: NON_VOTER last_known_addr { host: "127.6.61.60" port: 34607 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:04:26.013437 14360 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b9758e0d5bf44fa68a4085ab311c9f43 (127.6.61.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: NON_VOTER last_known_addr { host: "127.6.61.60" port: 34607 } attrs { promote: true } } }
I20260812 07:04:26.025667 14287 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.61.60:34607
I20260812 07:04:26.026615 14291 master.cc:537] Master@127.6.61.60:34607 shutting down...
I20260812 07:04:26.038364 14287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c86051a8e4146a3b059183c03351248" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: NON_VOTER last_known_addr { host: "127.6.61.60" port: 34607 } attrs { promote: true } } }
I20260812 07:04:26.038772 14287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:26.038578 14288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6c86051a8e4146a3b059183c03351248" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: NON_VOTER last_known_addr { host: "127.6.61.60" port: 34607 } attrs { promote: true } } }
I20260812 07:04:26.038951 14288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:26.039630 14391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c86051a8e4146a3b059183c03351248" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: NON_VOTER last_known_addr { host: "127.6.61.60" port: 34607 } attrs { promote: true } } }
I20260812 07:04:26.040022 14391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:04:26.043689 14454 raft_consensus.cc:858] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 34607 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: NON_VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } }
W20260812 07:04:26.066417 14429 tablet.cc:2369] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 07:04:26.044028 14454 raft_consensus.cc:1565] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 34607 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: NON_VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } }
I20260812 07:04:26.082526 14454 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 06a305d836de4590a62f589f803f0dfa (127.6.61.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: NON_VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: true } } }
I20260812 07:04:26.093761 14287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c86051a8e4146a3b059183c03351248" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: NON_VOTER last_known_addr { host: "127.6.61.60" port: 34607 } attrs { promote: true } } }
I20260812 07:04:26.094252 14287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:26.091488 14521 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c86051a8e4146a3b059183c03351248" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: NON_VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: true } } }
I20260812 07:04:26.098308 14521 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260812 07:04:26.098248 14493 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c86051a8e4146a3b059183c03351248. Latest consensus state: current_term: 1 leader_uuid: "6c86051a8e4146a3b059183c03351248" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: NON_VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: true } } }
I20260812 07:04:26.098562 14493 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [sys.catalog]: This master's current role is: NON_PARTICIPANT
W20260812 07:04:26.099529 13620 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260812 07:04:26.090965 14524 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c86051a8e4146a3b059183c03351248" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: NON_VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: true } } }
I20260812 07:04:26.099888 14524 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260812 07:04:26.101411 14291 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260812 07:04:26.102029 14291 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
W20260812 07:04:26.102039 13620 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 -> Peer b9758e0d5bf44fa68a4085ab311c9f43 (127.6.61.60:34607): Couldn't send request to peer b9758e0d5bf44fa68a4085ab311c9f43. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 07:04:26.102221 14291 tablet_replica.cc:330] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43: stopping tablet replica
I20260812 07:04:26.110651 14291 master.cc:552] Master@127.6.61.60:34607 shutdown complete.
I20260812 07:04:26.110802 14291 master_runner.cc:305] Clearing existing system tablet
I20260812 07:04:26.111274 14291 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:26.111482 14291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:26.114892 14291 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260812 07:04:26.116739 14528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:26.117205 14291 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:26.117699 14291 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 07:04:26.118007 14291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-2/wal
uuid: "b9758e0d5bf44fa68a4085ab311c9f43"
format_stamp: "Formatted at 2026-08-12 07:04:25 on dist-test-slave-69rg"
server_key: "3eea9695e87c34a4b43e82cc982ef5f9"
server_key_iv: "34f74048281814c0c318440c459e355b"
server_key_version: "encryptionkey@0"
I20260812 07:04:26.123627 14291 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:04:26.133610 14291 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 07:04:26.134187 14291 log.cc:1196] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260812 07:04:26.135067 14291 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43: Deleting consensus metadata
I20260812 07:04:26.135450 14291 master_runner.cc:315] Copying system tablet from 127.6.61.62:34449
I20260812 07:04:26.139086 14291 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43: tablet copy: Beginning tablet copy session from remote peer at address 127.6.61.62:34449
I20260812 07:04:26.161016 14534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:26.140128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.0.0.1:60082)
0812 07:04:26.141100 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:26.141125 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:26.141350 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:26.141878 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:26.141884 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:26.142576 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:26.143060 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:26.143077 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:26.144894 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:26.144901 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:26.147140 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:26.147176 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:26.148441 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:26.148459 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:26.148762 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:26.149994 (+ 1232us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:26.150033 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:26.151724 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:26.156849 (+ 5125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:26.156855 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:26.156860 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:26.157545 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:26.158151 (+ 606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:26.158157 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:26.158161 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:26.158469 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:26.159377 (+ 908us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:26.159383 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:26.159656 (+ 273us) client_negotiation.cc:769] Sending connection context
0812 07:04:26.159853 (+ 197us) client_negotiation.cc:240] Negotiation successful
0812 07:04:26.160386 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"thread_start_us":552,"threads_started":1}
I20260812 07:04:26.161054 14490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:26.140171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60082 (local address 127.6.61.62:34449)
0812 07:04:26.140447 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:26.140459 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:26.141226 (+ 767us) server_negotiation.cc:407] Connection header received
0812 07:04:26.141550 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:26.141557 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:26.141671 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:26.141805 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:26.143253 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:26.144684 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:26.147346 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:26.148177 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:26.148374 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:26.151992 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:26.151999 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:26.152005 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:26.152051 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:26.156684 (+ 4633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:26.157706 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:26.157715 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:26.157721 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:26.157994 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:26.158622 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:26.158629 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:26.158633 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:26.159206 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:26.159414 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:26.160105 (+ 691us) server_negotiation.cc:299] Negotiation successful
0812 07:04:26.160436 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 07:04:26.162551 13664 tablet_copy_service.cc:140] P 6c86051a8e4146a3b059183c03351248: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b9758e0d5bf44fa68a4085ab311c9f43 ({username='oryx', principal='oryx/127.6.61.60@KRBTEST.COM'} at 127.0.0.1:60082)
I20260812 07:04:26.162822 13664 tablet_copy_service.cc:161] P 6c86051a8e4146a3b059183c03351248: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b9758e0d5bf44fa68a4085ab311c9f43 at {username='oryx', principal='oryx/127.6.61.60@KRBTEST.COM'} at 127.0.0.1:60082: session id = b9758e0d5bf44fa68a4085ab311c9f43-00000000000000000000000000000000
I20260812 07:04:26.168015 13664 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 07:04:26.170463 14291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 07:04:26.179903 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 13709
I20260812 07:04:26.180827 14291 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43: tablet copy: Starting download of 0 data blocks...
I20260812 07:04:26.181181 14291 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43: tablet copy: Starting download of 1 WAL segments...
I20260812 07:04:26.184065 14291 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 07:04:26.189922 13664 tablet_copy_service.cc:342] P 6c86051a8e4146a3b059183c03351248: Request end of tablet copy session b9758e0d5bf44fa68a4085ab311c9f43-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.61.60@KRBTEST.COM'} at 127.0.0.1:60082
I20260812 07:04:26.190169 13664 tablet_copy_service.cc:433] P 6c86051a8e4146a3b059183c03351248: ending tablet copy session b9758e0d5bf44fa68a4085ab311c9f43-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b9758e0d5bf44fa68a4085ab311c9f43
I20260812 07:04:26.206185 14291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:26.212150 14536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:26.212926 14535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:26.214505 14538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:26.214777 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.1:42623
--local_ip_for_outbound_sockets=127.6.61.1
--tserver_master_addrs=127.6.61.62:34449
--webserver_port=44895
--webserver_interface=127.6.61.1
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.1.keytab
--principal=oryx/127.6.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:34261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.61.62:34449,127.6.61.61:39641
--tserver_master_addrs=127.6.61.62:34449,127.6.61.61:39641,127.6.61.60:34607 with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:04:26.218245 14291 server_base.cc:1034] running on GCE node
I20260812 07:04:26.218676 14291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:26.222107 14291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:26.223351 14291 hybrid_clock.cc:648] HybridClock initialized: now 1786518266223302 us; error 35 us; skew 500 ppm
I20260812 07:04:26.223683 14291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:04:26 dist-test-slave-69rg krb5kdc[13585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518266, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:26.226557 14291 init.cc:377] Logged in from keytab as oryx/127.6.61.60@KRBTEST.COM (short username oryx)
I20260812 07:04:26.228096 14291 webserver.cc:460] Webserver started at http://127.6.61.60:43641/ using document root <none> and password file <none>
I20260812 07:04:26.228490 14291 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:26.228605 14291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:26.232441 14291 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 07:04:26.234918 14547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:26.235463 14291 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 07:04:26.235612 14291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-2/wal
uuid: "b9758e0d5bf44fa68a4085ab311c9f43"
format_stamp: "Formatted at 2026-08-12 07:04:25 on dist-test-slave-69rg"
server_key: "3eea9695e87c34a4b43e82cc982ef5f9"
server_key_iv: "34f74048281814c0c318440c459e355b"
server_key_version: "encryptionkey@0"
I20260812 07:04:26.235832 14291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:26.271124 14291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:26.275880 14291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:26.304431 14291 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.60:34607
I20260812 07:04:26.304481 14601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.60:34607 every 8 connection(s)
I20260812 07:04:26.305665 14291 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/master-2/data/info.pb
I20260812 07:04:26.311856 14602 sys_catalog.cc:263] Verifying existing consensus state
I20260812 07:04:26.316519 14602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43: Bootstrap starting.
I20260812 07:04:26.333003 14602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 07:04:26.333590 14602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43: Bootstrap complete.
I20260812 07:04:26.334748 14602 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: NON_VOTER last_known_addr { host: "127.6.61.60" port: 34607 } attrs { promote: true } }
I20260812 07:04:26.335068 14602 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9758e0d5bf44fa68a4085ab311c9f43, State: Initialized, Role: LEARNER
I20260812 07:04:26.335369 14602 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: NON_VOTER last_known_addr { host: "127.6.61.60" port: 34607 } attrs { promote: true } }
I20260812 07:04:26.336305 14605 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: NON_VOTER last_known_addr { host: "127.6.61.60" port: 34607 } attrs { promote: true } } }
I20260812 07:04:26.336697 14602 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:04:26.336879 14605 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [sys.catalog]: This master's current role is: LEARNER
I20260812 07:04:26.350149 14617 catalog_manager.cc:1259] Loaded cluster ID: 71d35a330bec42eb92f085ce763b979e
I20260812 07:04:26.350265 14617 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43: loading cluster ID for follower catalog manager: success
I20260812 07:04:26.353751 14617 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43: acquiring CA information for follower catalog manager: success
I20260812 07:04:26.356313 14617 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 07:04:26.673785 14542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:26.674207 14542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:26.674377 14542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:26.684175 14542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:26.684358 14542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:26.684535 14542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.1
I20260812 07:04:26.697119 14542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:34261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.1.keytab
--principal=oryx/127.6.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.1:42623
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.61.1
--webserver_port=44895
--tserver_master_addrs=127.6.61.62:34449,127.6.61.61:39641,127.6.61.60:34607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.1
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:26.698778 14542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:26.700433 14542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 07:04:26.700697 14620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c86051a8e4146a3b059183c03351248" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: NON_VOTER last_known_addr { host: "127.6.61.60" port: 34607 } attrs { promote: true } } }
I20260812 07:04:26.701092 14620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:26.718189 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:26.700896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:34607 (local address 127.0.0.1:44676)
0812 07:04:26.709396 (+ 8500us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:26.709425 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:26.709629 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:26.711579 (+ 1950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:26.711585 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:26.711615 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:26.711860 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:26.711870 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:26.713703 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:26.713710 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:26.715757 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:26.715775 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:26.717279 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:26.717286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:26.717501 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:26.717538 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 07:04:26.717650 (+ 112us) client_negotiation.cc:240] Negotiation successful
0812 07:04:26.717842 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8305,"thread_start_us":8153,"threads_started":1}
I20260812 07:04:26.718715 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:26.701368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44676 (local address 127.6.61.60:34607)
0812 07:04:26.711131 (+ 9763us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:26.711141 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:26.711176 (+ 35us) server_negotiation.cc:407] Connection header received
0812 07:04:26.711274 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:26.711282 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:26.711404 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:26.711530 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:26.712035 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:26.713503 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:26.715949 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:26.717063 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:26.717203 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:26.717436 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:26.717728 (+ 292us) server_negotiation.cc:299] Negotiation successful
0812 07:04:26.717931 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9599,"thread_start_us":9408,"threads_started":1}
I20260812 07:04:26.719838 14575 raft_consensus.cc:1203] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260812 07:04:26.720546 14605 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c86051a8e4146a3b059183c03351248. Latest consensus state: current_term: 1 leader_uuid: "6c86051a8e4146a3b059183c03351248" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: NON_VOTER last_known_addr { host: "127.6.61.60" port: 34607 } attrs { promote: true } } }
I20260812 07:04:26.720916 14605 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [sys.catalog]: This master's current role is: LEARNER
I20260812 07:04:26.721271 14620 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248: attempting to promote NON_VOTER b9758e0d5bf44fa68a4085ab311c9f43 to VOTER
I20260812 07:04:26.722373 14620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 34607 } attrs { promote: false } }
W20260812 07:04:26.724673 14634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:26.725041 14635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:26.725409 14360 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 1 FOLLOWER]: Refusing update from remote peer 6c86051a8e4146a3b059183c03351248: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260812 07:04:26.725569 14542 server_base.cc:1034] running on GCE node
W20260812 07:04:26.725939 14637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:26.726398 14575 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [term 1 LEARNER]: Refusing update from remote peer 6c86051a8e4146a3b059183c03351248: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260812 07:04:26.726423 14626 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260812 07:04:26.726830 14542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:26.727113 14628 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 34607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260812 07:04:26.728379 14542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:26.729996 14626 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b9758e0d5bf44fa68a4085ab311c9f43 (127.6.61.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 34607 } attrs { promote: false } } }
I20260812 07:04:26.732357 14542 hybrid_clock.cc:648] HybridClock initialized: now 1786518266732054 us; error 254 us; skew 500 ppm
I20260812 07:04:26.732120 14575 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b9758e0d5bf44fa68a4085ab311c9f43 (127.6.61.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 34607 } attrs { promote: false } } }
I20260812 07:04:26.732993 14542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 07:04:26.733260 14360 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b9758e0d5bf44fa68a4085ab311c9f43 (127.6.61.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 34607 } attrs { promote: false } } }
I20260812 07:04:26.736615 14628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6c86051a8e4146a3b059183c03351248" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 34607 } attrs { promote: false } } }
I20260812 07:04:26.737077 14628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:26.737550 14628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c86051a8e4146a3b059183c03351248" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 34607 } attrs { promote: false } } }
I20260812 07:04:26.737890 14628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [sys.catalog]: This master's current role is: LEADER
Aug 12 07:04:26 dist-test-slave-69rg krb5kdc[13585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518266, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:26.738592 14605 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c86051a8e4146a3b059183c03351248" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 34607 } attrs { promote: false } } }
I20260812 07:04:26.738957 14605 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:04:26.738945 14628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c86051a8e4146a3b059183c03351248" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 34607 } attrs { promote: false } } }
I20260812 07:04:26.739332 14628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248 [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:26.740039 14542 init.cc:377] Logged in from keytab as oryx/127.6.61.1@KRBTEST.COM (short username oryx)
I20260812 07:04:26.742660 14642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c86051a8e4146a3b059183c03351248" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 34607 } attrs { promote: false } } }
I20260812 07:04:26.743126 14642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:04:26.743515 14542 webserver.cc:460] Webserver started at http://127.6.61.1:44895/ using document root <none> and password file <none>
I20260812 07:04:26.744436 14542 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:26.744624 14542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:26.751415 14542 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 07:04:26.755023 14648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:26.756531 14542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:04:26.756721 14542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-0/wal
uuid: "30fd1f2ae015406199f6b131c77b167f"
format_stamp: "Formatted at 2026-08-12 07:04:23 on dist-test-slave-69rg"
server_key: "abd074051d2d8197d0bdcee0b280ae92"
server_key_iv: "842ac178059b35faf287422b8c2d6889"
server_key_version: "encryptionkey@0"
I20260812 07:04:26.758093 14542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:26.789785 14542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:26.796613 14542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:26.797080 14542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:26.798832 14542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:26.800913 14542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:26.801112 14542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:26.801370 14542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:26.801568 14542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:26.876732 14542 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.1:42623
I20260812 07:04:26.876852 14770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.1:42623 every 8 connection(s)
I20260812 07:04:26.878618 14542 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
I20260812 07:04:26.882788 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 14542
I20260812 07:04:26.883195 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 13864
Aug 12 07:04:26 dist-test-slave-69rg krb5kdc[13585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518266, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.61@KRBTEST.COM
Aug 12 07:04:26 dist-test-slave-69rg krb5kdc[13585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.61.2@KRBTEST.COM
Aug 12 07:04:26 dist-test-slave-69rg krb5kdc[13585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518266, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM
Aug 12 07:04:26 dist-test-slave-69rg krb5kdc[13585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518266, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.60@KRBTEST.COM
I20260812 07:04:26.933746 14782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:26.885708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:33297 (local address 127.6.61.61:39641)
0812 07:04:26.902443 (+ 16735us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:26.902454 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:26.902484 (+ 30us) server_negotiation.cc:407] Connection header received
0812 07:04:26.906331 (+ 3847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:26.906335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:26.906440 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:26.906531 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:26.911033 (+ 4502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:26.912070 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:26.915777 (+ 3707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:26.916727 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:26.916891 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:26.923340 (+ 6449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:26.923346 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:26.923350 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:26.923386 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:26.928869 (+ 5483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:26.929911 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:26.929917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:26.929920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:26.930096 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:26.930620 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:26.930625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:26.930628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:26.931172 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:26.931346 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:26.932726 (+ 1380us) server_negotiation.cc:299] Negotiation successful
0812 07:04:26.933012 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16476,"thread_start_us":16794,"threads_started":1}
I20260812 07:04:26.936594 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:26.884124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:37497 (local address 127.6.61.60:34607)
0812 07:04:26.884410 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:26.884419 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:26.897448 (+ 13029us) server_negotiation.cc:407] Connection header received
0812 07:04:26.907015 (+ 9567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:26.907023 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:26.907137 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:26.907227 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:26.913235 (+ 6008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:26.914802 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:26.919803 (+ 5001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:26.920914 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:26.921049 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:26.926890 (+ 5841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:26.926899 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:26.926913 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:26.926961 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:26.932039 (+ 5078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:26.933101 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:26.933109 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:26.933116 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:26.933426 (+ 310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:26.934346 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:26.934352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:26.934356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:26.934823 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:26.934967 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:26.935707 (+ 740us) server_negotiation.cc:299] Negotiation successful
0812 07:04:26.936045 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260812 07:04:26.937049 14776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:26.884345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:39641 (local address 127.6.61.1:33297)
0812 07:04:26.901735 (+ 17390us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:26.901758 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:26.906138 (+ 4380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:26.906860 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:26.906864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:26.909136 (+ 2272us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:26.910798 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:26.910827 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:26.912310 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:26.912315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:26.915555 (+ 3240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:26.915582 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:26.917439 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:26.917446 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:26.917677 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:26.918473 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:26.918497 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:26.923037 (+ 4540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:26.929026 (+ 5989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:26.929037 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:26.929057 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:26.929747 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:26.930227 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:26.930232 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:26.930236 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:26.930487 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:26.931492 (+ 1005us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:26.931503 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:26.932051 (+ 548us) client_negotiation.cc:769] Sending connection context
0812 07:04:26.934146 (+ 2095us) client_negotiation.cc:240] Negotiation successful
0812 07:04:26.934795 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17189,"mutex_wait_us":30,"spinlock_wait_cycles":16128,"thread_start_us":13132,"threads_started":1}
I20260812 07:04:26.938570 14784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:26.904152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.6.61.1:57367)
0812 07:04:26.905652 (+ 1500us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:26.905670 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:26.905852 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:26.906775 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:26.906790 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:26.909368 (+ 2578us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:26.910806 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:26.910827 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:26.914390 (+ 3563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:26.914397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:26.916748 (+ 2351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:26.916764 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:26.919555 (+ 2791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:26.919564 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:26.919822 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:26.920704 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:26.920731 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:26.925481 (+ 4750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:26.933902 (+ 8421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:26.933910 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:26.933914 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:26.934579 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:26.935130 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:26.935138 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:26.935145 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:26.935461 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:26.936474 (+ 1013us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:26.936479 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:26.936635 (+ 156us) client_negotiation.cc:769] Sending connection context
0812 07:04:26.936884 (+ 249us) client_negotiation.cc:240] Negotiation successful
0812 07:04:26.937366 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1327,"mutex_wait_us":278,"thread_start_us":1086,"threads_started":1}
I20260812 07:04:26.939333 14780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:26.885212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36934 (local address 127.6.61.2:39417)
0812 07:04:26.895764 (+ 10552us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:26.895778 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:26.903757 (+ 7979us) server_negotiation.cc:407] Connection header received
0812 07:04:26.904052 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:26.904063 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:26.904439 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:26.905372 (+ 933us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:26.908244 (+ 2872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:26.909704 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:26.912578 (+ 2874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:26.913807 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:26.914043 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:26.926440 (+ 12397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:26.926466 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:26.926488 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:26.926562 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:26.933632 (+ 7070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:26.934868 (+ 1236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:26.934882 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:26.934888 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:26.935170 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:26.935799 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:26.935804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:26.935807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:26.936373 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:26.936819 (+ 446us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:26.938309 (+ 1490us) server_negotiation.cc:299] Negotiation successful
0812 07:04:26.938685 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10253,"thread_start_us":9951,"threads_started":1}
I20260812 07:04:26.939366 14773 heartbeater.cc:347] Connected to a master server at 127.6.61.61:39641
I20260812 07:04:26.939886 14773 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:26.939944 14783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:26.902119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:57367 (local address 127.6.61.62:34449)
0812 07:04:26.905057 (+ 2938us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:26.905064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:26.905928 (+ 864us) server_negotiation.cc:407] Connection header received
0812 07:04:26.906196 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:26.906202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:26.906291 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:26.906444 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:26.911066 (+ 4622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:26.912861 (+ 1795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:26.917954 (+ 5093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:26.919242 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:26.919399 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:26.925891 (+ 6492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:26.925900 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:26.925904 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:26.925966 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:26.933034 (+ 7068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:26.934741 (+ 1707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:26.934746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:26.934750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:26.934980 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:26.935643 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:26.935648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:26.935652 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:26.936174 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:26.936339 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:26.939046 (+ 2707us) server_negotiation.cc:299] Negotiation successful
0812 07:04:26.939307 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2716,"thread_start_us":2510,"threads_started":1}
I20260812 07:04:26.939182 14778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:26.883536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:34607 (local address 127.6.61.1:37497)
0812 07:04:26.897193 (+ 13657us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:26.897266 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:26.905852 (+ 8586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:26.907556 (+ 1704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:26.907561 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:26.909164 (+ 1603us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:26.910803 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:26.910832 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:26.915108 (+ 4276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:26.915115 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:26.917725 (+ 2610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:26.917744 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:26.921658 (+ 3914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:26.921670 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:26.922096 (+ 426us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:26.922972 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:26.922998 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:26.926521 (+ 3523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:26.932240 (+ 5719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:26.932246 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:26.932250 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:26.932939 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:26.933564 (+ 625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:26.933568 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:26.933570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:26.934179 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:26.935084 (+ 905us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:26.935088 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:26.935331 (+ 243us) client_negotiation.cc:769] Sending connection context
0812 07:04:26.937475 (+ 2144us) client_negotiation.cc:240] Negotiation successful
0812 07:04:26.937874 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13089,"thread_start_us":17144,"threads_started":1}
I20260812 07:04:26.940869 14773 heartbeater.cc:511] Master 127.6.61.61:39641 requested a full tablet report, sending...
I20260812 07:04:26.941017 14012 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:26.942325 14772 heartbeater.cc:347] Connected to a master server at 127.6.61.62:34449
I20260812 07:04:26.942662 14771 heartbeater.cc:347] Connected to a master server at 127.6.61.60:34607
I20260812 07:04:26.942871 14771 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:26.943101 14772 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:26.943971 14771 heartbeater.cc:511] Master 127.6.61.60:34607 requested a full tablet report, sending...
I20260812 07:04:26.945600 14350 ts_manager.cc:194] Registered new tserver with Master: 30fd1f2ae015406199f6b131c77b167f (127.6.61.1:42623)
I20260812 07:04:26.946592 14772 heartbeater.cc:511] Master 127.6.61.62:34449 requested a full tablet report, sending...
I20260812 07:04:26.946555 14565 ts_manager.cc:194] Registered new tserver with Master: 30fd1f2ae015406199f6b131c77b167f (127.6.61.1:42623)
I20260812 07:04:26.948160 13639 ts_manager.cc:194] Re-registered known tserver with Master: 30fd1f2ae015406199f6b131c77b167f (127.6.61.1:42623)
I20260812 07:04:26.949038 13639 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.61.1@KRBTEST.COM'} at 127.6.61.1:57367
I20260812 07:04:27.042949 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 13864
W20260812 07:04:27.068416 13622 connection.cc:535] server connection from 127.6.61.2:35425 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 07:04:27.069206 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.2:39417
--local_ip_for_outbound_sockets=127.6.61.2
--tserver_master_addrs=127.6.61.62:34449
--webserver_port=39463
--webserver_interface=127.6.61.2
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.2.keytab
--principal=oryx/127.6.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:34261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.61.62:34449,127.6.61.61:39641
--tserver_master_addrs=127.6.61.62:34449,127.6.61.61:39641,127.6.61.60:34607 with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:27.414489 14786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:27.414772 14786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:27.414932 14786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:27.422365 14786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:27.422489 14786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:27.422676 14786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.2
I20260812 07:04:27.433406 14786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:34261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.2.keytab
--principal=oryx/127.6.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.2:39417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.61.2
--webserver_port=39463
--tserver_master_addrs=127.6.61.62:34449,127.6.61.61:39641,127.6.61.60:34607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.2
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:27.435045 14786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:27.436578 14786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:27.453637 14792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:27.453608 14793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:27.454161 14795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:27.454643 14786 server_base.cc:1034] running on GCE node
I20260812 07:04:27.455302 14786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:27.456280 14786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:27.457528 14786 hybrid_clock.cc:648] HybridClock initialized: now 1786518267457477 us; error 62 us; skew 500 ppm
I20260812 07:04:27.457937 14786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:04:27 dist-test-slave-69rg krb5kdc[13585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518267, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:27.461536 14786 init.cc:377] Logged in from keytab as oryx/127.6.61.2@KRBTEST.COM (short username oryx)
I20260812 07:04:27.463327 14786 webserver.cc:460] Webserver started at http://127.6.61.2:39463/ using document root <none> and password file <none>
I20260812 07:04:27.464063 14786 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:27.464241 14786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:27.470045 14786 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 07:04:27.472874 14802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:27.474473 14786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:04:27.474675 14786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-1/wal
uuid: "d6f37996431d4b60afeb49f20e3fbba4"
format_stamp: "Formatted at 2026-08-12 07:04:23 on dist-test-slave-69rg"
server_key: "bf17441fb24d90c969a0e5a1b909e697"
server_key_iv: "eec2a9197c0fa72a96d9116a95fffd71"
server_key_version: "encryptionkey@0"
I20260812 07:04:27.475862 14786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:27.508198 14786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:27.512519 14786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:27.512850 14786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:27.513983 14786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:27.515450 14786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:27.515573 14786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:27.515710 14786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:27.515811 14786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:27.551710 14786 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.2:39417
I20260812 07:04:27.551808 14940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.2:39417 every 8 connection(s)
I20260812 07:04:27.553215 14786 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
I20260812 07:04:27.556610 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 14786
I20260812 07:04:27.556921 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 14023
Aug 12 07:04:27 dist-test-slave-69rg krb5kdc[13585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.61.3@KRBTEST.COM
Aug 12 07:04:27 dist-test-slave-69rg krb5kdc[13585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518267, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.61@KRBTEST.COM
Aug 12 07:04:27 dist-test-slave-69rg krb5kdc[13585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518267, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.60@KRBTEST.COM
Aug 12 07:04:27 dist-test-slave-69rg krb5kdc[13585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518267, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM
I20260812 07:04:27.598575 14948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:27.557568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:40565 (local address 127.6.61.61:39641)
0812 07:04:27.568381 (+ 10813us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:27.568391 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:27.569779 (+ 1388us) server_negotiation.cc:407] Connection header received
0812 07:04:27.573114 (+ 3335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:27.573125 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:27.573228 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:27.573509 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:27.576968 (+ 3459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:27.578151 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:27.580829 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:27.581473 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:27.581617 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:27.588575 (+ 6958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:27.588582 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:27.588587 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:27.588632 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:27.593757 (+ 5125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:27.595084 (+ 1327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:27.595093 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:27.595099 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:27.595375 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:27.596096 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:27.596103 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:27.596107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:27.596596 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:27.596794 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:27.597766 (+ 972us) server_negotiation.cc:299] Negotiation successful
0812 07:04:27.598055 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10632,"thread_start_us":10821,"threads_started":1}
I20260812 07:04:27.599153 14951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:27.558931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35016 (local address 127.6.61.3:34313)
0812 07:04:27.568419 (+ 9488us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:27.568427 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:27.569093 (+ 666us) server_negotiation.cc:407] Connection header received
0812 07:04:27.569463 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:27.569487 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:27.570481 (+ 994us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:27.570849 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:27.573066 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:27.576709 (+ 3643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:27.578690 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:27.579563 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:27.579694 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:27.587338 (+ 7644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:27.587351 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:27.587364 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:27.587428 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:27.593201 (+ 5773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:27.594410 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:27.594422 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:27.594428 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:27.594708 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:27.595421 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:27.595428 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:27.595432 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:27.596041 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:27.596438 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:27.598137 (+ 1699us) server_negotiation.cc:299] Negotiation successful
0812 07:04:27.598518 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9308,"thread_start_us":9090,"threads_started":1}
I20260812 07:04:27.600616 14945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:27.556254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:34431 (local address 127.6.61.60:34607)
0812 07:04:27.556857 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:27.556863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:27.573932 (+ 17069us) server_negotiation.cc:407] Connection header received
0812 07:04:27.574772 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:27.574780 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:27.574894 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:27.575157 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:27.577693 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:27.579808 (+ 2115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:27.583364 (+ 3556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:27.583975 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:27.584109 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:27.588796 (+ 4687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:27.588805 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:27.588812 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:27.588882 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:27.596215 (+ 7333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:27.597471 (+ 1256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:27.597478 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:27.597483 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:27.597686 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:27.598348 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:27.598354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:27.598358 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:27.598865 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:27.599050 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:27.599807 (+ 757us) server_negotiation.cc:299] Negotiation successful
0812 07:04:27.600050 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":360,"threads_started":1}
I20260812 07:04:27.601038 14166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:27.601634 14947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:27.557289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:34607 (local address 127.6.61.2:34431)
0812 07:04:27.573681 (+ 16392us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:27.573700 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:27.574524 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:27.575459 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:27.575466 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:27.577306 (+ 1840us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:27.577517 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:27.577525 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:27.579961 (+ 2436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:27.579966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:27.583171 (+ 3205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:27.583184 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:27.584174 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:27.584182 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:27.584356 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:27.585143 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:27.585199 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:27.588523 (+ 3324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:27.596414 (+ 7891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:27.596419 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:27.596422 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:27.597296 (+ 874us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:27.597879 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:27.597887 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:27.597891 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:27.598159 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:27.599195 (+ 1036us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:27.599201 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:27.599372 (+ 171us) client_negotiation.cc:769] Sending connection context
0812 07:04:27.600769 (+ 1397us) client_negotiation.cc:240] Negotiation successful
0812 07:04:27.600974 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16236,"mutex_wait_us":228,"spinlock_wait_cycles":58752,"thread_start_us":15916,"threads_started":1}
I20260812 07:04:27.602087 14946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:27.556980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:39641 (local address 127.6.61.2:40565)
0812 07:04:27.569477 (+ 12497us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:27.569535 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:27.572790 (+ 3255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:27.574574 (+ 1784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:27.574589 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:27.575908 (+ 1319us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:27.576773 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:27.576803 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:27.578355 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:27.578361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:27.580614 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:27.580631 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:27.581954 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:27.581965 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:27.582403 (+ 438us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:27.583148 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:27.583184 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:27.588227 (+ 5043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:27.593939 (+ 5712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:27.593950 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:27.593970 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:27.594908 (+ 938us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:27.595560 (+ 652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:27.595565 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:27.595569 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:27.595896 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:27.596899 (+ 1003us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:27.596909 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:27.597339 (+ 430us) client_negotiation.cc:769] Sending connection context
0812 07:04:27.597665 (+ 326us) client_negotiation.cc:240] Negotiation successful
0812 07:04:27.598383 (+ 718us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12117,"mutex_wait_us":58,"thread_start_us":11737,"threads_started":1}
I20260812 07:04:27.603878 14943 heartbeater.cc:347] Connected to a master server at 127.6.61.61:39641
I20260812 07:04:27.603896 14941 heartbeater.cc:347] Connected to a master server at 127.6.61.60:34607
I20260812 07:04:27.604323 14943 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:27.604365 14941 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:27.605010 14953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:27.574015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.6.61.2:57205)
0812 07:04:27.576136 (+ 2121us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:27.576157 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:27.576393 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:27.577725 (+ 1332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:27.577730 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:27.578325 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:27.578675 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:27.578688 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:27.579987 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:27.579991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:27.582198 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:27.582226 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:27.584818 (+ 2592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:27.584831 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:27.585047 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:27.585935 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:27.585962 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:27.592170 (+ 6208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:27.598990 (+ 6820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:27.598994 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:27.598997 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:27.599709 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:27.601569 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:27.601577 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:27.601580 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:27.601925 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:27.603451 (+ 1526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:27.603456 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:27.603613 (+ 157us) client_negotiation.cc:769] Sending connection context
0812 07:04:27.603943 (+ 330us) client_negotiation.cc:240] Negotiation successful
0812 07:04:27.604302 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1960,"mutex_wait_us":109,"thread_start_us":1764,"threads_started":1}
I20260812 07:04:27.605540 14952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:27.574089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:57205 (local address 127.6.61.62:34449)
0812 07:04:27.574638 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:27.574647 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:27.576369 (+ 1722us) server_negotiation.cc:407] Connection header received
0812 07:04:27.576517 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:27.576524 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:27.576609 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:27.576859 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:27.578825 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:27.579687 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:27.582345 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:27.582784 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:27.582929 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:27.592434 (+ 9505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:27.592440 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:27.592443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:27.592480 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:27.596936 (+ 4456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:27.599857 (+ 2921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:27.599863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:27.599867 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:27.600117 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:27.602768 (+ 2651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:27.602774 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:27.602777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:27.603317 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:27.603482 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:27.604584 (+ 1102us) server_negotiation.cc:299] Negotiation successful
0812 07:04:27.604901 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":263,"threads_started":1}
I20260812 07:04:27.605736 14941 heartbeater.cc:511] Master 127.6.61.60:34607 requested a full tablet report, sending...
I20260812 07:04:27.606598 14942 heartbeater.cc:347] Connected to a master server at 127.6.61.62:34449
I20260812 07:04:27.606792 14942 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:27.607892 14942 heartbeater.cc:511] Master 127.6.61.62:34449 requested a full tablet report, sending...
I20260812 07:04:27.608021 14943 heartbeater.cc:511] Master 127.6.61.61:39641 requested a full tablet report, sending...
I20260812 07:04:27.608453 14565 ts_manager.cc:194] Registered new tserver with Master: d6f37996431d4b60afeb49f20e3fbba4 (127.6.61.2:39417)
I20260812 07:04:27.609320 13639 ts_manager.cc:194] Re-registered known tserver with Master: d6f37996431d4b60afeb49f20e3fbba4 (127.6.61.2:39417)
I20260812 07:04:27.609851 14350 ts_manager.cc:194] Registered new tserver with Master: d6f37996431d4b60afeb49f20e3fbba4 (127.6.61.2:39417)
I20260812 07:04:27.610710 13639 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.61.2@KRBTEST.COM'} at 127.6.61.2:57205
I20260812 07:04:27.702508 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 14023
I20260812 07:04:27.724262 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.3:34313
--local_ip_for_outbound_sockets=127.6.61.3
--tserver_master_addrs=127.6.61.62:34449
--webserver_port=38245
--webserver_interface=127.6.61.3
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.3.keytab
--principal=oryx/127.6.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:34261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.61.62:34449,127.6.61.61:39641
--tserver_master_addrs=127.6.61.62:34449,127.6.61.61:39641,127.6.61.60:34607 with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:04:27.951742 14772 heartbeater.cc:503] Master 127.6.61.62:34449 was elected leader, sending a full tablet report...
W20260812 07:04:28.082530 14955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:28.082799 14955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:28.082896 14955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:28.090463 14955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:28.090572 14955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:28.090745 14955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.3
I20260812 07:04:28.100750 14955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:34261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.3.keytab
--principal=oryx/127.6.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.3:34313
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.61.3
--webserver_port=38245
--tserver_master_addrs=127.6.61.62:34449,127.6.61.61:39641,127.6.61.60:34607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.3
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:28.102408 14955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:28.103794 14955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:28.119128 14964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:28.119186 14962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:28.119211 14961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:28.120733 14955 server_base.cc:1034] running on GCE node
I20260812 07:04:28.121421 14955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:28.122488 14955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:28.123724 14955 hybrid_clock.cc:648] HybridClock initialized: now 1786518268123673 us; error 33 us; skew 500 ppm
I20260812 07:04:28.124073 14955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:04:28 dist-test-slave-69rg krb5kdc[13585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518268, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:28.127506 14955 init.cc:377] Logged in from keytab as oryx/127.6.61.3@KRBTEST.COM (short username oryx)
I20260812 07:04:28.129024 14955 webserver.cc:460] Webserver started at http://127.6.61.3:38245/ using document root <none> and password file <none>
I20260812 07:04:28.129750 14955 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:28.129941 14955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:28.136015 14955 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 07:04:28.139144 14972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:28.140551 14955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:04:28.140728 14955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-2/wal
uuid: "1a5f51df7c0c43bc85366d1c15dc2f89"
format_stamp: "Formatted at 2026-08-12 07:04:24 on dist-test-slave-69rg"
server_key: "648e331102b7840000c5bfdb606abd59"
server_key_iv: "dc7192be850187f577207fe7bf7b9397"
server_key_version: "encryptionkey@0"
I20260812 07:04:28.141775 14955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:28.183209 14955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:28.187973 14955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:28.188328 14955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:28.189379 14955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:28.190929 14955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:28.191066 14955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:28.191200 14955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:28.191294 14955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:28.227532 14955 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.3:34313
I20260812 07:04:28.227630 15099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.3:34313 every 8 connection(s)
I20260812 07:04:28.229187 14955 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
I20260812 07:04:28.232481 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 14955
Aug 12 07:04:28 dist-test-slave-69rg krb5kdc[13585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518268, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.60@KRBTEST.COM
Aug 12 07:04:28 dist-test-slave-69rg krb5kdc[13585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518268, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.61@KRBTEST.COM
I20260812 07:04:28.283125 15105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:28.233108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:34607 (local address 127.6.61.3:60971)
0812 07:04:28.240850 (+ 7742us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:28.240894 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:28.244184 (+ 3290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:28.245589 (+ 1405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:28.245607 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:28.246693 (+ 1086us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:28.247926 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:28.247959 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.250279 (+ 2320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.250288 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:28.253205 (+ 2917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:28.253226 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.255073 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.255085 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:28.256448 (+ 1363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:28.258192 (+ 1744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:28.258273 (+ 81us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:28.263748 (+ 5475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:28.272951 (+ 9203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:28.272965 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:28.272994 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:28.273917 (+ 923us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:28.274636 (+ 719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:28.274645 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:28.274651 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:28.275094 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:28.278958 (+ 3864us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:28.278971 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:28.279640 (+ 669us) client_negotiation.cc:769] Sending connection context
0812 07:04:28.280045 (+ 405us) client_negotiation.cc:240] Negotiation successful
0812 07:04:28.280878 (+ 833us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7437,"mutex_wait_us":3,"spinlock_wait_cycles":17280,"thread_start_us":7196,"threads_started":1}
I20260812 07:04:28.284806 15110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:28.234368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:39641 (local address 127.6.61.3:34187)
0812 07:04:28.246721 (+ 12353us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:28.246745 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:28.247219 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:28.248123 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:28.248129 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:28.248982 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:28.249378 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:28.249394 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.251253 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.251261 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:28.253371 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:28.253391 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.255578 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.255589 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:28.256031 (+ 442us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:28.258817 (+ 2786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:28.258858 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:28.267751 (+ 8893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:28.277375 (+ 9624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:28.277383 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:28.277389 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:28.278391 (+ 1002us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:28.281597 (+ 3206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:28.281606 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:28.281612 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:28.281903 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:28.282997 (+ 1094us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:28.283005 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:28.283249 (+ 244us) client_negotiation.cc:769] Sending connection context
0812 07:04:28.283467 (+ 218us) client_negotiation.cc:240] Negotiation successful
0812 07:04:28.283918 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12090,"mutex_wait_us":2,"spinlock_wait_cycles":3840,"thread_start_us":24798,"threads_started":1}
I20260812 07:04:28.286279 15106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:28.233906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:60971 (local address 127.6.61.60:34607)
0812 07:04:28.243000 (+ 9094us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:28.243009 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:28.243040 (+ 31us) server_negotiation.cc:407] Connection header received
0812 07:04:28.244776 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:28.244785 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:28.244907 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:28.245035 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:28.248210 (+ 3175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.250018 (+ 1808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.253419 (+ 3401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.254390 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.254557 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:28.264160 (+ 9603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:28.264194 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:28.264202 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:28.264251 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:28.271124 (+ 6873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:28.274176 (+ 3052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:28.274197 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:28.274203 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:28.274471 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:28.278189 (+ 3718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:28.278197 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:28.278204 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:28.278766 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:28.284658 (+ 5892us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:28.284872 (+ 214us) server_negotiation.cc:299] Negotiation successful
0812 07:04:28.285554 (+ 682us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8928,"thread_start_us":8813,"threads_started":1}
I20260812 07:04:28.286614 15104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:28.232301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:34187 (local address 127.6.61.61:39641)
0812 07:04:28.246093 (+ 13792us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:28.246103 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:28.247026 (+ 923us) server_negotiation.cc:407] Connection header received
0812 07:04:28.247401 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:28.247410 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:28.247517 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:28.247778 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:28.249593 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.251031 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.253588 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.254795 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.254936 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:28.268011 (+ 13075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:28.268030 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:28.268036 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:28.268118 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:28.277144 (+ 9026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:28.280558 (+ 3414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:28.280568 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:28.280575 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:28.281395 (+ 820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:28.282081 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:28.282090 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:28.282096 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:28.282650 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:28.282887 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:28.285437 (+ 2550us) server_negotiation.cc:299] Negotiation successful
0812 07:04:28.285767 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13461,"thread_start_us":13535,"threads_started":1}
I20260812 07:04:28.289516 15102 heartbeater.cc:347] Connected to a master server at 127.6.61.61:39641
I20260812 07:04:28.290112 15102 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:28.291757 15102 heartbeater.cc:511] Master 127.6.61.61:39641 requested a full tablet report, sending...
I20260812 07:04:28.293066 15100 heartbeater.cc:347] Connected to a master server at 127.6.61.60:34607
I20260812 07:04:28.293254 15100 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:28.295606 14350 ts_manager.cc:194] Registered new tserver with Master: 1a5f51df7c0c43bc85366d1c15dc2f89 (127.6.61.3:34313)
I20260812 07:04:28.297515 15100 heartbeater.cc:511] Master 127.6.61.60:34607 requested a full tablet report, sending...
Aug 12 07:04:28 dist-test-slave-69rg krb5kdc[13585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518268, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM
I20260812 07:04:28.299341 14565 ts_manager.cc:194] Registered new tserver with Master: 1a5f51df7c0c43bc85366d1c15dc2f89 (127.6.61.3:34313)
I20260812 07:04:28.310333 15109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:28.242260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:54479 (local address 127.6.61.62:34449)
0812 07:04:28.255953 (+ 13693us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:28.255962 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:28.260690 (+ 4728us) server_negotiation.cc:407] Connection header received
0812 07:04:28.260997 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:28.261004 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:28.261112 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:28.261402 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:28.272780 (+ 11378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.274666 (+ 1886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.278604 (+ 3938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.279996 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.280184 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:28.300390 (+ 20206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:28.300411 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:28.300419 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:28.300492 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:28.306301 (+ 5809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:28.307711 (+ 1410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:28.307718 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:28.307723 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:28.308024 (+ 301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:28.308729 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:28.308734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:28.308738 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:28.309018 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:28.309175 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:28.309699 (+ 524us) server_negotiation.cc:299] Negotiation successful
0812 07:04:28.309968 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13502,"thread_start_us":13346,"threads_started":1}
I20260812 07:04:28.310452 15108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:28.245782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.6.61.3:54479)
0812 07:04:28.260450 (+ 14668us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:28.260477 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:28.260809 (+ 332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:28.261478 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:28.261487 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:28.263277 (+ 1790us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:28.272567 (+ 9290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:28.272591 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.275286 (+ 2695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.275295 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:28.277909 (+ 2614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:28.277930 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.287022 (+ 9092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.287036 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:28.287468 (+ 432us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:28.289789 (+ 2321us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:28.289874 (+ 85us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:28.300107 (+ 10233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:28.306881 (+ 6774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:28.306887 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:28.306891 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:28.307549 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:28.308232 (+ 683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:28.308237 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:28.308241 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:28.308483 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:28.309175 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:28.309180 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:28.309337 (+ 157us) client_negotiation.cc:769] Sending connection context
0812 07:04:28.309465 (+ 128us) client_negotiation.cc:240] Negotiation successful
0812 07:04:28.309810 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14404,"thread_start_us":1075,"threads_started":1}
I20260812 07:04:28.311546 15101 heartbeater.cc:347] Connected to a master server at 127.6.61.62:34449
I20260812 07:04:28.311771 15101 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:28.313871 15101 heartbeater.cc:511] Master 127.6.61.62:34449 requested a full tablet report, sending...
I20260812 07:04:28.315474 13639 ts_manager.cc:194] Re-registered known tserver with Master: 1a5f51df7c0c43bc85366d1c15dc2f89 (127.6.61.3:34313)
I20260812 07:04:28.317039 13639 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.61.3@KRBTEST.COM'} at 127.6.61.3:54479
I20260812 07:04:28.331152 15114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:28.260493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60084 (local address 127.6.61.62:34449)
0812 07:04:28.268591 (+ 8098us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:28.268597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:28.289475 (+ 20878us) server_negotiation.cc:407] Connection header received
0812 07:04:28.292039 (+ 2564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:28.292048 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:28.292194 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:28.292409 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:28.301420 (+ 9011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.302931 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.311585 (+ 8654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.312581 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.312728 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:28.320538 (+ 7810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:28.320545 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:28.320549 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:28.320592 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:28.326320 (+ 5728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:28.327646 (+ 1326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:28.327661 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:28.327666 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:28.327916 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:28.328600 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:28.328605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:28.328609 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:28.329219 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:28.329587 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:28.330123 (+ 536us) server_negotiation.cc:299] Negotiation successful
0812 07:04:28.330474 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7900,"thread_start_us":9865,"threads_started":1}
I20260812 07:04:28.336413 15104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:28.267518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60786 (local address 127.6.61.61:39641)
0812 07:04:28.297543 (+ 30025us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:28.297549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:28.301630 (+ 4081us) server_negotiation.cc:407] Connection header received
0812 07:04:28.303742 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:28.303747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:28.303871 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:28.303960 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:28.317733 (+ 13773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.319422 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.321571 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.322445 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.322623 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:28.326392 (+ 3769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:28.326397 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:28.326401 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:28.326485 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:28.331322 (+ 4837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:28.332976 (+ 1654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:28.332983 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:28.332986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:28.333346 (+ 360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:28.334062 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:28.334067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:28.334069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:28.334498 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:28.334977 (+ 479us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:28.335418 (+ 441us) server_negotiation.cc:299] Negotiation successful
0812 07:04:28.335695 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29833,"thread_start_us":29628,"threads_started":1}
I20260812 07:04:28.340517 15106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:28.290054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44686 (local address 127.6.61.60:34607)
0812 07:04:28.290420 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:28.290428 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:28.302511 (+ 12083us) server_negotiation.cc:407] Connection header received
0812 07:04:28.303740 (+ 1229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:28.303745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:28.303859 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:28.303937 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:28.317462 (+ 13525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.318713 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.323444 (+ 4731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.324456 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.324596 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:28.329196 (+ 4600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:28.329205 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:28.329209 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:28.329267 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:28.336074 (+ 6807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:28.337172 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:28.337178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:28.337181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:28.337504 (+ 323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:28.338145 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:28.338151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:28.338154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:28.338684 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:28.339052 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:28.339892 (+ 840us) server_negotiation.cc:299] Negotiation successful
0812 07:04:28.340135 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205}
I20260812 07:04:28.336918 13639 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:04:28.348932 13639 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:04:28.396374 15133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:28.377041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:34313 (local address 127.0.0.1:35028)
0812 07:04:28.379753 (+ 2712us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:28.379772 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:28.380147 (+ 375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:28.382010 (+ 1863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:28.382019 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:28.382058 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:28.382852 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:28.382873 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.388224 (+ 5351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.388234 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:28.392206 (+ 3972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:28.392222 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.395041 (+ 2819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.395049 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:28.395311 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:28.395347 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 07:04:28.395466 (+ 119us) client_negotiation.cc:240] Negotiation successful
0812 07:04:28.395628 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2528,"thread_start_us":2179,"threads_started":1}
I20260812 07:04:28.397567 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:28.377413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35028 (local address 127.6.61.3:34313)
0812 07:04:28.379013 (+ 1600us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:28.379027 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:28.380326 (+ 1299us) server_negotiation.cc:407] Connection header received
0812 07:04:28.380517 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:28.380529 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:28.380909 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:28.381342 (+ 433us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:28.383403 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.387918 (+ 4515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.392499 (+ 4581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.394757 (+ 2258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.394962 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:28.395312 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:28.396774 (+ 1462us) server_negotiation.cc:299] Negotiation successful
0812 07:04:28.397029 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1283,"thread_start_us":920,"threads_started":1}
I20260812 07:04:28.402573 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:28.375305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:39417 (local address 127.0.0.1:36948)
0812 07:04:28.377062 (+ 1757us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:28.377089 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:28.377483 (+ 394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:28.392353 (+ 14870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:28.392360 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:28.392392 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:28.393380 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:28.393395 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.395662 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.395667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:28.399027 (+ 3360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:28.399041 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.401456 (+ 2415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.401464 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:28.401743 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:28.401778 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 07:04:28.401881 (+ 103us) client_negotiation.cc:240] Negotiation successful
0812 07:04:28.402011 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1522,"thread_start_us":1211,"threads_started":1}
I20260812 07:04:28.403007 15028 tablet_service.cc:1467] Processing CreateTablet for tablet 464f6cc6f086450eb76cdcb0831ca4d1 (DEFAULT_TABLE table=test-table [id=54a2d6b29841494c8bc6c3beb0968b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:04:28.403923 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:28.375479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36948 (local address 127.6.61.2:39417)
0812 07:04:28.390956 (+ 15477us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:28.390966 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:28.391001 (+ 35us) server_negotiation.cc:407] Connection header received
0812 07:04:28.391173 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:28.391192 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:28.391589 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:28.393693 (+ 2104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:28.394102 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.395512 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.399267 (+ 3755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.401100 (+ 1833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.402965 (+ 1865us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:28.403269 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:28.403342 (+ 73us) server_negotiation.cc:299] Negotiation successful
0812 07:04:28.403547 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15238,"thread_start_us":16517,"threads_started":1}
I20260812 07:04:28.404608 15136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:28.380100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58176 (local address 127.6.61.1:42623)
0812 07:04:28.391073 (+ 10973us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:28.391081 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:28.392686 (+ 1605us) server_negotiation.cc:407] Connection header received
0812 07:04:28.393185 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:28.393203 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:28.393690 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:28.394116 (+ 426us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:28.395421 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.396984 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.401297 (+ 4313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.403070 (+ 1773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.403152 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:28.403334 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:28.404108 (+ 774us) server_negotiation.cc:299] Negotiation successful
0812 07:04:28.404322 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10762,"thread_start_us":10999,"threads_started":1}
I20260812 07:04:28.404910 15028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 464f6cc6f086450eb76cdcb0831ca4d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:28.405686 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:28.379700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:42623 (local address 127.0.0.1:58176)
0812 07:04:28.392368 (+ 12668us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:28.392389 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:28.392850 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:28.394112 (+ 1262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:28.394118 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:28.394147 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:28.395035 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:28.395051 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.397151 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.397156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:28.401007 (+ 3851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:28.401020 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.403652 (+ 2632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.403660 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:28.403885 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:28.403920 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 07:04:28.404907 (+ 987us) client_negotiation.cc:240] Negotiation successful
0812 07:04:28.405064 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12468,"thread_start_us":12132,"threads_started":1}
I20260812 07:04:28.408116 14859 tablet_service.cc:1467] Processing CreateTablet for tablet 464f6cc6f086450eb76cdcb0831ca4d1 (DEFAULT_TABLE table=test-table [id=54a2d6b29841494c8bc6c3beb0968b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:04:28.409245 14859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 464f6cc6f086450eb76cdcb0831ca4d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:28.410259 14702 tablet_service.cc:1467] Processing CreateTablet for tablet 464f6cc6f086450eb76cdcb0831ca4d1 (DEFAULT_TABLE table=test-table [id=54a2d6b29841494c8bc6c3beb0968b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:04:28.412067 14702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 464f6cc6f086450eb76cdcb0831ca4d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:28.425457 15137 tablet_bootstrap.cc:492] T 464f6cc6f086450eb76cdcb0831ca4d1 P d6f37996431d4b60afeb49f20e3fbba4: Bootstrap starting.
I20260812 07:04:28.431274 15138 tablet_bootstrap.cc:492] T 464f6cc6f086450eb76cdcb0831ca4d1 P 1a5f51df7c0c43bc85366d1c15dc2f89: Bootstrap starting.
I20260812 07:04:28.431650 15137 tablet_bootstrap.cc:654] T 464f6cc6f086450eb76cdcb0831ca4d1 P d6f37996431d4b60afeb49f20e3fbba4: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:28.434702 15137 log.cc:824] T 464f6cc6f086450eb76cdcb0831ca4d1 P d6f37996431d4b60afeb49f20e3fbba4: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:28.434870 15140 tablet_bootstrap.cc:492] T 464f6cc6f086450eb76cdcb0831ca4d1 P 30fd1f2ae015406199f6b131c77b167f: Bootstrap starting.
I20260812 07:04:28.438212 15138 tablet_bootstrap.cc:654] T 464f6cc6f086450eb76cdcb0831ca4d1 P 1a5f51df7c0c43bc85366d1c15dc2f89: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:28.440006 15138 log.cc:824] T 464f6cc6f086450eb76cdcb0831ca4d1 P 1a5f51df7c0c43bc85366d1c15dc2f89: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:28.441995 15137 tablet_bootstrap.cc:492] T 464f6cc6f086450eb76cdcb0831ca4d1 P d6f37996431d4b60afeb49f20e3fbba4: No bootstrap required, opened a new log
I20260812 07:04:28.442509 15137 ts_tablet_manager.cc:1397] T 464f6cc6f086450eb76cdcb0831ca4d1 P d6f37996431d4b60afeb49f20e3fbba4: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.000s
I20260812 07:04:28.443007 15140 tablet_bootstrap.cc:654] T 464f6cc6f086450eb76cdcb0831ca4d1 P 30fd1f2ae015406199f6b131c77b167f: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:28.445298 15140 log.cc:824] T 464f6cc6f086450eb76cdcb0831ca4d1 P 30fd1f2ae015406199f6b131c77b167f: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:28.449370 15138 tablet_bootstrap.cc:492] T 464f6cc6f086450eb76cdcb0831ca4d1 P 1a5f51df7c0c43bc85366d1c15dc2f89: No bootstrap required, opened a new log
I20260812 07:04:28.449720 15138 ts_tablet_manager.cc:1397] T 464f6cc6f086450eb76cdcb0831ca4d1 P 1a5f51df7c0c43bc85366d1c15dc2f89: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.013s
I20260812 07:04:28.455603 15140 tablet_bootstrap.cc:492] T 464f6cc6f086450eb76cdcb0831ca4d1 P 30fd1f2ae015406199f6b131c77b167f: No bootstrap required, opened a new log
I20260812 07:04:28.456025 15140 ts_tablet_manager.cc:1397] T 464f6cc6f086450eb76cdcb0831ca4d1 P 30fd1f2ae015406199f6b131c77b167f: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.006s
I20260812 07:04:28.461475 15137 raft_consensus.cc:348] T 464f6cc6f086450eb76cdcb0831ca4d1 P d6f37996431d4b60afeb49f20e3fbba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f37996431d4b60afeb49f20e3fbba4" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 39417 } } peers { permanent_uuid: "1a5f51df7c0c43bc85366d1c15dc2f89" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 34313 } } peers { permanent_uuid: "30fd1f2ae015406199f6b131c77b167f" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42623 } }
I20260812 07:04:28.462472 15137 raft_consensus.cc:374] T 464f6cc6f086450eb76cdcb0831ca4d1 P d6f37996431d4b60afeb49f20e3fbba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:28.462671 15137 raft_consensus.cc:729] T 464f6cc6f086450eb76cdcb0831ca4d1 P d6f37996431d4b60afeb49f20e3fbba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6f37996431d4b60afeb49f20e3fbba4, State: Initialized, Role: FOLLOWER
I20260812 07:04:28.463560 15137 consensus_queue.cc:260] T 464f6cc6f086450eb76cdcb0831ca4d1 P d6f37996431d4b60afeb49f20e3fbba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f37996431d4b60afeb49f20e3fbba4" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 39417 } } peers { permanent_uuid: "1a5f51df7c0c43bc85366d1c15dc2f89" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 34313 } } peers { permanent_uuid: "30fd1f2ae015406199f6b131c77b167f" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42623 } }
I20260812 07:04:28.463620 15138 raft_consensus.cc:348] T 464f6cc6f086450eb76cdcb0831ca4d1 P 1a5f51df7c0c43bc85366d1c15dc2f89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f37996431d4b60afeb49f20e3fbba4" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 39417 } } peers { permanent_uuid: "1a5f51df7c0c43bc85366d1c15dc2f89" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 34313 } } peers { permanent_uuid: "30fd1f2ae015406199f6b131c77b167f" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42623 } }
I20260812 07:04:28.464087 15138 raft_consensus.cc:374] T 464f6cc6f086450eb76cdcb0831ca4d1 P 1a5f51df7c0c43bc85366d1c15dc2f89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:28.464237 15138 raft_consensus.cc:729] T 464f6cc6f086450eb76cdcb0831ca4d1 P 1a5f51df7c0c43bc85366d1c15dc2f89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a5f51df7c0c43bc85366d1c15dc2f89, State: Initialized, Role: FOLLOWER
I20260812 07:04:28.465255 15138 consensus_queue.cc:260] T 464f6cc6f086450eb76cdcb0831ca4d1 P 1a5f51df7c0c43bc85366d1c15dc2f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f37996431d4b60afeb49f20e3fbba4" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 39417 } } peers { permanent_uuid: "1a5f51df7c0c43bc85366d1c15dc2f89" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 34313 } } peers { permanent_uuid: "30fd1f2ae015406199f6b131c77b167f" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42623 } }
I20260812 07:04:28.466629 15137 ts_tablet_manager.cc:1428] T 464f6cc6f086450eb76cdcb0831ca4d1 P d6f37996431d4b60afeb49f20e3fbba4: Time spent starting tablet: real 0.024s user 0.018s sys 0.004s
I20260812 07:04:28.467103 14942 heartbeater.cc:503] Master 127.6.61.62:34449 was elected leader, sending a full tablet report...
I20260812 07:04:28.469314 15138 ts_tablet_manager.cc:1428] T 464f6cc6f086450eb76cdcb0831ca4d1 P 1a5f51df7c0c43bc85366d1c15dc2f89: Time spent starting tablet: real 0.019s user 0.004s sys 0.014s
I20260812 07:04:28.469769 15101 heartbeater.cc:503] Master 127.6.61.62:34449 was elected leader, sending a full tablet report...
I20260812 07:04:28.476377 15140 raft_consensus.cc:348] T 464f6cc6f086450eb76cdcb0831ca4d1 P 30fd1f2ae015406199f6b131c77b167f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f37996431d4b60afeb49f20e3fbba4" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 39417 } } peers { permanent_uuid: "1a5f51df7c0c43bc85366d1c15dc2f89" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 34313 } } peers { permanent_uuid: "30fd1f2ae015406199f6b131c77b167f" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42623 } }
I20260812 07:04:28.477407 15140 raft_consensus.cc:374] T 464f6cc6f086450eb76cdcb0831ca4d1 P 30fd1f2ae015406199f6b131c77b167f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:28.477596 15140 raft_consensus.cc:729] T 464f6cc6f086450eb76cdcb0831ca4d1 P 30fd1f2ae015406199f6b131c77b167f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30fd1f2ae015406199f6b131c77b167f, State: Initialized, Role: FOLLOWER
I20260812 07:04:28.478752 15140 consensus_queue.cc:260] T 464f6cc6f086450eb76cdcb0831ca4d1 P 30fd1f2ae015406199f6b131c77b167f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f37996431d4b60afeb49f20e3fbba4" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 39417 } } peers { permanent_uuid: "1a5f51df7c0c43bc85366d1c15dc2f89" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 34313 } } peers { permanent_uuid: "30fd1f2ae015406199f6b131c77b167f" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42623 } }
I20260812 07:04:28.481125 15140 ts_tablet_manager.cc:1428] T 464f6cc6f086450eb76cdcb0831ca4d1 P 30fd1f2ae015406199f6b131c77b167f: Time spent starting tablet: real 0.025s user 0.014s sys 0.007s
W20260812 07:04:28.489436 15103 tablet.cc:2369] T 464f6cc6f086450eb76cdcb0831ca4d1 P 1a5f51df7c0c43bc85366d1c15dc2f89: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 07:04:28.558195 14944 tablet.cc:2369] T 464f6cc6f086450eb76cdcb0831ca4d1 P d6f37996431d4b60afeb49f20e3fbba4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 07:04:28.634902 14774 tablet.cc:2369] T 464f6cc6f086450eb76cdcb0831ca4d1 P 30fd1f2ae015406199f6b131c77b167f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:04:28.741595 15144 raft_consensus.cc:482] T 464f6cc6f086450eb76cdcb0831ca4d1 P d6f37996431d4b60afeb49f20e3fbba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:04:28.742152 15144 raft_consensus.cc:504] T 464f6cc6f086450eb76cdcb0831ca4d1 P d6f37996431d4b60afeb49f20e3fbba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f37996431d4b60afeb49f20e3fbba4" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 39417 } } peers { permanent_uuid: "1a5f51df7c0c43bc85366d1c15dc2f89" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 34313 } } peers { permanent_uuid: "30fd1f2ae015406199f6b131c77b167f" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42623 } }
I20260812 07:04:28.744462 15144 leader_election.cc:288] T 464f6cc6f086450eb76cdcb0831ca4d1 P d6f37996431d4b60afeb49f20e3fbba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a5f51df7c0c43bc85366d1c15dc2f89 (127.6.61.3:34313), 30fd1f2ae015406199f6b131c77b167f (127.6.61.1:42623)
I20260812 07:04:28.754676 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:28.744895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:45671 (local address 127.6.61.3:34313)
0812 07:04:28.745073 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:28.745078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:28.746819 (+ 1741us) server_negotiation.cc:407] Connection header received
0812 07:04:28.747254 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:28.747258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:28.747332 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:28.747437 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:28.748604 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.749590 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.752269 (+ 2679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.753386 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.753557 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:28.753731 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:28.754181 (+ 450us) server_negotiation.cc:299] Negotiation successful
0812 07:04:28.754311 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 07:04:28.754968 15136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:28.746764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:49943 (local address 127.6.61.1:42623)
0812 07:04:28.747064 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:28.747072 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:28.747788 (+ 716us) server_negotiation.cc:407] Connection header received
0812 07:04:28.747957 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:28.747963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:28.748065 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:28.748215 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:28.749048 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.750053 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.752608 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.753695 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.753907 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:28.754050 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:28.754450 (+ 400us) server_negotiation.cc:299] Negotiation successful
0812 07:04:28.754580 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260812 07:04:28.755012 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:28.745004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:34313 (local address 127.6.61.2:45671)
0812 07:04:28.746547 (+ 1543us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:28.746568 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:28.747007 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:28.747502 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:28.747509 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:28.747553 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:28.748438 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:28.748459 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.749769 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.749774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:28.752055 (+ 2281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:28.752069 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.753716 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.753723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:28.754010 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:28.754036 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 07:04:28.754153 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 07:04:28.754288 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1288,"mutex_wait_us":1,"thread_start_us":857,"threads_started":1}
I20260812 07:04:28.755507 15149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:28.746339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:42623 (local address 127.6.61.2:49943)
0812 07:04:28.747560 (+ 1221us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:28.747574 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:28.747809 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:28.748228 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:28.748232 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:28.748250 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:28.748831 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:28.748840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.750196 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.750203 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:28.752440 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:28.752454 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.754144 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.754151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:28.754282 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:28.754304 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 07:04:28.754421 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 07:04:28.754553 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1101,"mutex_wait_us":23,"thread_start_us":921,"threads_started":1}
I20260812 07:04:28.756582 15050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464f6cc6f086450eb76cdcb0831ca4d1" candidate_uuid: "d6f37996431d4b60afeb49f20e3fbba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a5f51df7c0c43bc85366d1c15dc2f89" is_pre_election: true
I20260812 07:04:28.757153 14724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464f6cc6f086450eb76cdcb0831ca4d1" candidate_uuid: "d6f37996431d4b60afeb49f20e3fbba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30fd1f2ae015406199f6b131c77b167f" is_pre_election: true
I20260812 07:04:28.757608 15050 raft_consensus.cc:2454] T 464f6cc6f086450eb76cdcb0831ca4d1 P 1a5f51df7c0c43bc85366d1c15dc2f89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6f37996431d4b60afeb49f20e3fbba4 in term 0.
I20260812 07:04:28.758023 14724 raft_consensus.cc:2454] T 464f6cc6f086450eb76cdcb0831ca4d1 P 30fd1f2ae015406199f6b131c77b167f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6f37996431d4b60afeb49f20e3fbba4 in term 0.
I20260812 07:04:28.758599 14807 leader_election.cc:302] T 464f6cc6f086450eb76cdcb0831ca4d1 P d6f37996431d4b60afeb49f20e3fbba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a5f51df7c0c43bc85366d1c15dc2f89, d6f37996431d4b60afeb49f20e3fbba4; no voters:
I20260812 07:04:28.759215 15144 raft_consensus.cc:2781] T 464f6cc6f086450eb76cdcb0831ca4d1 P d6f37996431d4b60afeb49f20e3fbba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:04:28.759537 15144 raft_consensus.cc:482] T 464f6cc6f086450eb76cdcb0831ca4d1 P d6f37996431d4b60afeb49f20e3fbba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:04:28.759665 15144 raft_consensus.cc:3037] T 464f6cc6f086450eb76cdcb0831ca4d1 P d6f37996431d4b60afeb49f20e3fbba4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:28.763015 15144 raft_consensus.cc:504] T 464f6cc6f086450eb76cdcb0831ca4d1 P d6f37996431d4b60afeb49f20e3fbba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f37996431d4b60afeb49f20e3fbba4" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 39417 } } peers { permanent_uuid: "1a5f51df7c0c43bc85366d1c15dc2f89" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 34313 } } peers { permanent_uuid: "30fd1f2ae015406199f6b131c77b167f" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42623 } }
I20260812 07:04:28.764014 15144 leader_election.cc:288] T 464f6cc6f086450eb76cdcb0831ca4d1 P d6f37996431d4b60afeb49f20e3fbba4 [CANDIDATE]: Term 1 election: Requested vote from peers 1a5f51df7c0c43bc85366d1c15dc2f89 (127.6.61.3:34313), 30fd1f2ae015406199f6b131c77b167f (127.6.61.1:42623)
I20260812 07:04:28.764631 15050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464f6cc6f086450eb76cdcb0831ca4d1" candidate_uuid: "d6f37996431d4b60afeb49f20e3fbba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a5f51df7c0c43bc85366d1c15dc2f89"
I20260812 07:04:28.764719 14724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464f6cc6f086450eb76cdcb0831ca4d1" candidate_uuid: "d6f37996431d4b60afeb49f20e3fbba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30fd1f2ae015406199f6b131c77b167f"
I20260812 07:04:28.764935 15050 raft_consensus.cc:3037] T 464f6cc6f086450eb76cdcb0831ca4d1 P 1a5f51df7c0c43bc85366d1c15dc2f89 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:28.765000 14724 raft_consensus.cc:3037] T 464f6cc6f086450eb76cdcb0831ca4d1 P 30fd1f2ae015406199f6b131c77b167f [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:28.768389 15050 raft_consensus.cc:2454] T 464f6cc6f086450eb76cdcb0831ca4d1 P 1a5f51df7c0c43bc85366d1c15dc2f89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6f37996431d4b60afeb49f20e3fbba4 in term 1.
I20260812 07:04:28.768860 14724 raft_consensus.cc:2454] T 464f6cc6f086450eb76cdcb0831ca4d1 P 30fd1f2ae015406199f6b131c77b167f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6f37996431d4b60afeb49f20e3fbba4 in term 1.
I20260812 07:04:28.769100 14807 leader_election.cc:302] T 464f6cc6f086450eb76cdcb0831ca4d1 P d6f37996431d4b60afeb49f20e3fbba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a5f51df7c0c43bc85366d1c15dc2f89, d6f37996431d4b60afeb49f20e3fbba4; no voters:
I20260812 07:04:28.769518 15144 raft_consensus.cc:2781] T 464f6cc6f086450eb76cdcb0831ca4d1 P d6f37996431d4b60afeb49f20e3fbba4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:04:28.770460 15144 raft_consensus.cc:686] T 464f6cc6f086450eb76cdcb0831ca4d1 P d6f37996431d4b60afeb49f20e3fbba4 [term 1 LEADER]: Becoming Leader. State: Replica: d6f37996431d4b60afeb49f20e3fbba4, State: Running, Role: LEADER
I20260812 07:04:28.771256 15144 consensus_queue.cc:237] T 464f6cc6f086450eb76cdcb0831ca4d1 P d6f37996431d4b60afeb49f20e3fbba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f37996431d4b60afeb49f20e3fbba4" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 39417 } } peers { permanent_uuid: "1a5f51df7c0c43bc85366d1c15dc2f89" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 34313 } } peers { permanent_uuid: "30fd1f2ae015406199f6b131c77b167f" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42623 } }
I20260812 07:04:28.779006 13638 catalog_manager.cc:5495] T 464f6cc6f086450eb76cdcb0831ca4d1 P d6f37996431d4b60afeb49f20e3fbba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6f37996431d4b60afeb49f20e3fbba4 (127.6.61.2). New cstate: current_term: 1 leader_uuid: "d6f37996431d4b60afeb49f20e3fbba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f37996431d4b60afeb49f20e3fbba4" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a5f51df7c0c43bc85366d1c15dc2f89" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 34313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30fd1f2ae015406199f6b131c77b167f" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42623 } health_report { overall_health: UNKNOWN } } }
I20260812 07:04:28.868997 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:28.862068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36962 (local address 127.6.61.2:39417)
0812 07:04:28.862337 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:28.862343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:28.862606 (+ 263us) server_negotiation.cc:407] Connection header received
0812 07:04:28.862782 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:28.862786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:28.862871 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:28.863013 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:04:28.863619 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.864425 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.866409 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.866902 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.867010 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:28.867754 (+ 744us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:04:28.868310 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:04:28.868485 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:28.868604 (+ 119us) server_negotiation.cc:299] Negotiation successful
0812 07:04:28.868693 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260812 07:04:28.880028 15050 raft_consensus.cc:1261] T 464f6cc6f086450eb76cdcb0831ca4d1 P 1a5f51df7c0c43bc85366d1c15dc2f89 [term 1 FOLLOWER]: Refusing update from remote peer d6f37996431d4b60afeb49f20e3fbba4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:04:28.880081 14724 raft_consensus.cc:1261] T 464f6cc6f086450eb76cdcb0831ca4d1 P 30fd1f2ae015406199f6b131c77b167f [term 1 FOLLOWER]: Refusing update from remote peer d6f37996431d4b60afeb49f20e3fbba4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:04:28.881354 15151 consensus_queue.cc:1035] T 464f6cc6f086450eb76cdcb0831ca4d1 P d6f37996431d4b60afeb49f20e3fbba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30fd1f2ae015406199f6b131c77b167f" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:04:28.881933 15144 consensus_queue.cc:1035] T 464f6cc6f086450eb76cdcb0831ca4d1 P d6f37996431d4b60afeb49f20e3fbba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a5f51df7c0c43bc85366d1c15dc2f89" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:04:28.899144 15156 mvcc.cc:205] Tried to move back new op lower bound from 7317578829319356416 to 7317578828891660288. Current Snapshot: MvccSnapshot[applied={T|T < 7317578829319356416}]
I20260812 07:04:28.902349 15158 mvcc.cc:205] Tried to move back new op lower bound from 7317578829319356416 to 7317578828891660288. Current Snapshot: MvccSnapshot[applied={T|T < 7317578829319356416}]
I20260812 07:04:28.912367 15157 mvcc.cc:205] Tried to move back new op lower bound from 7317578829319356416 to 7317578828891660288. Current Snapshot: MvccSnapshot[applied={T|T < 7317578829319356416}]
I20260812 07:04:28.924186 13638 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:04:28.924558 13638 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:04:28.931743 13638 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c86051a8e4146a3b059183c03351248: Sending DeleteTablet for 3 replicas of tablet 464f6cc6f086450eb76cdcb0831ca4d1
I20260812 07:04:28.933781 14859 tablet_service.cc:1514] Processing DeleteTablet for tablet 464f6cc6f086450eb76cdcb0831ca4d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:04:28 UTC) from {username='oryx', principal='oryx/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:36948
I20260812 07:04:28.934293 14702 tablet_service.cc:1514] Processing DeleteTablet for tablet 464f6cc6f086450eb76cdcb0831ca4d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:04:28 UTC) from {username='oryx', principal='oryx/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:58176
I20260812 07:04:28.935186 15028 tablet_service.cc:1514] Processing DeleteTablet for tablet 464f6cc6f086450eb76cdcb0831ca4d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:04:28 UTC) from {username='oryx', principal='oryx/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:35028
I20260812 07:04:28.936010 15167 tablet_replica.cc:330] T 464f6cc6f086450eb76cdcb0831ca4d1 P d6f37996431d4b60afeb49f20e3fbba4: stopping tablet replica
I20260812 07:04:28.936189 15169 tablet_replica.cc:330] T 464f6cc6f086450eb76cdcb0831ca4d1 P 30fd1f2ae015406199f6b131c77b167f: stopping tablet replica
I20260812 07:04:28.936750 15169 raft_consensus.cc:2229] T 464f6cc6f086450eb76cdcb0831ca4d1 P 30fd1f2ae015406199f6b131c77b167f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:04:28.936879 15167 raft_consensus.cc:2229] T 464f6cc6f086450eb76cdcb0831ca4d1 P d6f37996431d4b60afeb49f20e3fbba4 [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:04:28.937712 15169 raft_consensus.cc:2258] T 464f6cc6f086450eb76cdcb0831ca4d1 P 30fd1f2ae015406199f6b131c77b167f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:04:28.938247 15167 raft_consensus.cc:2258] T 464f6cc6f086450eb76cdcb0831ca4d1 P d6f37996431d4b60afeb49f20e3fbba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:04:28.940270 15169 ts_tablet_manager.cc:1905] T 464f6cc6f086450eb76cdcb0831ca4d1 P 30fd1f2ae015406199f6b131c77b167f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:04:28.940276 15167 ts_tablet_manager.cc:1905] T 464f6cc6f086450eb76cdcb0831ca4d1 P d6f37996431d4b60afeb49f20e3fbba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:04:28.946168 15170 tablet_replica.cc:330] T 464f6cc6f086450eb76cdcb0831ca4d1 P 1a5f51df7c0c43bc85366d1c15dc2f89: stopping tablet replica
I20260812 07:04:28.946808 15170 raft_consensus.cc:2229] T 464f6cc6f086450eb76cdcb0831ca4d1 P 1a5f51df7c0c43bc85366d1c15dc2f89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:04:28.947613 15170 raft_consensus.cc:2258] T 464f6cc6f086450eb76cdcb0831ca4d1 P 1a5f51df7c0c43bc85366d1c15dc2f89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:04:28.949997 15170 ts_tablet_manager.cc:1905] T 464f6cc6f086450eb76cdcb0831ca4d1 P 1a5f51df7c0c43bc85366d1c15dc2f89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:04:28.950697 15167 ts_tablet_manager.cc:1918] T 464f6cc6f086450eb76cdcb0831ca4d1 P d6f37996431d4b60afeb49f20e3fbba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:04:28.951037 15167 log.cc:1196] T 464f6cc6f086450eb76cdcb0831ca4d1 P d6f37996431d4b60afeb49f20e3fbba4: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-1/wal/wals/464f6cc6f086450eb76cdcb0831ca4d1
I20260812 07:04:28.952087 15167 ts_tablet_manager.cc:1939] T 464f6cc6f086450eb76cdcb0831ca4d1 P d6f37996431d4b60afeb49f20e3fbba4: Deleting consensus metadata
I20260812 07:04:28.954736 13624 catalog_manager.cc:4841] TS d6f37996431d4b60afeb49f20e3fbba4 (127.6.61.2:39417): tablet 464f6cc6f086450eb76cdcb0831ca4d1 (table test-table [id=54a2d6b29841494c8bc6c3beb0968b07]) successfully deleted
I20260812 07:04:28.956668 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 13600
I20260812 07:04:28.962554 15169 ts_tablet_manager.cc:1918] T 464f6cc6f086450eb76cdcb0831ca4d1 P 30fd1f2ae015406199f6b131c77b167f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:04:28.962805 15169 log.cc:1196] T 464f6cc6f086450eb76cdcb0831ca4d1 P 30fd1f2ae015406199f6b131c77b167f: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-0/wal/wals/464f6cc6f086450eb76cdcb0831ca4d1
I20260812 07:04:28.963794 15169 ts_tablet_manager.cc:1939] T 464f6cc6f086450eb76cdcb0831ca4d1 P 30fd1f2ae015406199f6b131c77b167f: Deleting consensus metadata
I20260812 07:04:28.963810 15170 ts_tablet_manager.cc:1918] T 464f6cc6f086450eb76cdcb0831ca4d1 P 1a5f51df7c0c43bc85366d1c15dc2f89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:04:28.964037 15170 log.cc:1196] T 464f6cc6f086450eb76cdcb0831ca4d1 P 1a5f51df7c0c43bc85366d1c15dc2f89: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-2/wal/wals/464f6cc6f086450eb76cdcb0831ca4d1
I20260812 07:04:28.965349 15170 ts_tablet_manager.cc:1939] T 464f6cc6f086450eb76cdcb0831ca4d1 P 1a5f51df7c0c43bc85366d1c15dc2f89: Deleting consensus metadata
I20260812 07:04:28.965992 13674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:28.968475 13624 catalog_manager.cc:4841] TS 1a5f51df7c0c43bc85366d1c15dc2f89 (127.6.61.3:34313): tablet 464f6cc6f086450eb76cdcb0831ca4d1 (table test-table [id=54a2d6b29841494c8bc6c3beb0968b07]) successfully deleted
I20260812 07:04:28.994709 15173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:28.944520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60798 (local address 127.6.61.61:39641)
0812 07:04:28.955726 (+ 11206us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:28.955735 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:28.955767 (+ 32us) server_negotiation.cc:407] Connection header received
0812 07:04:28.955863 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:28.955868 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:28.955967 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:28.956154 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:28.968797 (+ 12643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.970780 (+ 1983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.973448 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:28.974189 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:28.974271 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:28.985183 (+ 10912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:28.985190 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:28.985195 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:28.985248 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:28.990241 (+ 4993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:28.991572 (+ 1331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:28.991580 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:28.991585 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:28.991817 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:28.992437 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:28.992444 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:28.992447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:28.992947 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:28.993261 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:28.993804 (+ 543us) server_negotiation.cc:299] Negotiation successful
0812 07:04:28.994103 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11063,"thread_start_us":12507,"threads_started":1}
I20260812 07:04:29.018392 15174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:28.955679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44698 (local address 127.6.61.60:34607)
0812 07:04:28.972677 (+ 16998us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:28.972684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:28.994751 (+ 22067us) server_negotiation.cc:407] Connection header received
0812 07:04:28.995207 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:28.995214 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:28.995314 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:28.995409 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:28.999328 (+ 3919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:29.000690 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:29.003313 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:29.004111 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:29.004289 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:29.009140 (+ 4851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:29.009149 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:29.009154 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:29.009201 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:29.014051 (+ 4850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:29.015299 (+ 1248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:29.015316 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:29.015320 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:29.015532 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:29.016102 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:29.016107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:29.016111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:29.016615 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:29.016901 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:29.017456 (+ 555us) server_negotiation.cc:299] Negotiation successful
0812 07:04:29.017738 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16825,"thread_start_us":17011,"threads_started":1}
I20260812 07:04:29.110987 13624 catalog_manager.cc:4841] TS 30fd1f2ae015406199f6b131c77b167f (127.6.61.1:42623): tablet 464f6cc6f086450eb76cdcb0831ca4d1 (table test-table [id=54a2d6b29841494c8bc6c3beb0968b07]) successfully deleted
I20260812 07:04:29.111698 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 13600
I20260812 07:04:29.172817 15173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:29.152459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60814 (local address 127.6.61.61:39641)
0812 07:04:29.152719 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:29.152731 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:29.153812 (+ 1081us) server_negotiation.cc:407] Connection header received
0812 07:04:29.154255 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:29.154263 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:29.154359 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:29.154441 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:29.157042 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:29.158362 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:29.160196 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:29.160813 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:29.160900 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:29.164869 (+ 3969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:29.164876 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:29.164884 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:29.164929 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:29.169103 (+ 4174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:29.170041 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:29.170048 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:29.170052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:29.170210 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:29.170706 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:29.170712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:29.170715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:29.171092 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:29.171390 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:29.172104 (+ 714us) server_negotiation.cc:299] Negotiation successful
0812 07:04:29.172350 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 07:04:29.173365 15174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:29.153714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44700 (local address 127.6.61.60:34607)
0812 07:04:29.154087 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:29.154094 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:29.155684 (+ 1590us) server_negotiation.cc:407] Connection header received
0812 07:04:29.155793 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:29.155797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:29.155870 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:29.155997 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:29.158350 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:29.159455 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:29.161262 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:29.162093 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:29.162252 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:29.165597 (+ 3345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:29.165608 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:29.165616 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:29.165669 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:29.169692 (+ 4023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:29.170598 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:29.170607 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:29.170611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:29.170775 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:29.171214 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:29.171219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:29.171223 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:29.171599 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:29.171888 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:29.172426 (+ 538us) server_negotiation.cc:299] Negotiation successful
0812 07:04:29.172658 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195}
I20260812 07:04:29.910847 15184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:29.910050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.6.61.2:60571)
0812 07:04:29.910549 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:29.910679 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":249,"threads_started":1}
W20260812 07:04:29.911397 14942 heartbeater.cc:644] Failed to heartbeat to 127.6.61.62:34449 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
I20260812 07:04:29.916268 15185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:29.915382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.6.61.1:40701)
0812 07:04:29.915900 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:29.916090 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":223,"threads_started":1}
W20260812 07:04:29.916769 14772 heartbeater.cc:644] Failed to heartbeat to 127.6.61.62:34449 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
I20260812 07:04:29.919662 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:29.918724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.6.61.3:55933)
0812 07:04:29.919293 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:29.919442 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":286,"threads_started":1}
W20260812 07:04:29.920070 15101 heartbeater.cc:644] Failed to heartbeat to 127.6.61.62:34449 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
I20260812 07:04:30.431980 15187 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6c86051a8e4146a3b059183c03351248)
I20260812 07:04:30.432255 15187 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 34607 } attrs { promote: false } }
I20260812 07:04:30.433038 15189 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6c86051a8e4146a3b059183c03351248)
I20260812 07:04:30.433249 15189 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 34607 } attrs { promote: false } }
I20260812 07:04:30.434010 15187 leader_election.cc:288] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6c86051a8e4146a3b059183c03351248 (127.6.61.62:34449), 06a305d836de4590a62f589f803f0dfa (127.6.61.61:39641)
I20260812 07:04:30.435290 15191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:30.434118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.0.0.1:37726)
0812 07:04:30.434806 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:30.435014 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":357,"threads_started":1}
W20260812 07:04:30.435570 14548 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
I20260812 07:04:30.435801 15189 leader_election.cc:288] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6c86051a8e4146a3b059183c03351248 (127.6.61.62:34449), b9758e0d5bf44fa68a4085ab311c9f43 (127.6.61.60:34607)
I20260812 07:04:30.436669 15194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:30.434469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.0.0.1:37732)
0812 07:04:30.436018 (+ 1549us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:30.436324 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1400,"thread_start_us":256,"threads_started":1}
I20260812 07:04:30.438123 15191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:30.436875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.0.0.1:37736)
0812 07:04:30.437780 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:30.437893 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":743}
W20260812 07:04:30.438069 14329 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
W20260812 07:04:30.438463 14548 leader_election.cc:334] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6c86051a8e4146a3b059183c03351248 (127.6.61.62:34449): Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
I20260812 07:04:30.440850 15198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:30.440098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.0.0.1:37742)
0812 07:04:30.440352 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:30.440574 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260812 07:04:30.441059 14329 leader_election.cc:334] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6c86051a8e4146a3b059183c03351248 (127.6.61.62:34449): Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
I20260812 07:04:30.445030 15194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:30.436850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:34607 (local address 127.0.0.1:52158)
0812 07:04:30.437033 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:30.437055 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:30.437211 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:30.438289 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:30.438293 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:30.438318 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:30.438602 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:30.438617 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:30.440434 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:30.440450 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:30.442798 (+ 2348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:30.442816 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:30.444131 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:30.444137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:30.444354 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:30.444382 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 07:04:30.444521 (+ 139us) client_negotiation.cc:240] Negotiation successful
0812 07:04:30.444673 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":24,"thread_start_us":901,"threads_started":1}
I20260812 07:04:30.445024 15199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:30.437110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52158 (local address 127.6.61.60:34607)
0812 07:04:30.437898 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:30.437904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:30.437930 (+ 26us) server_negotiation.cc:407] Connection header received
0812 07:04:30.438007 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:30.438011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:30.438138 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:30.438761 (+ 623us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:30.439001 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:30.440274 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:30.443027 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:30.443915 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:30.444070 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:30.444179 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:30.444558 (+ 379us) server_negotiation.cc:299] Negotiation successful
0812 07:04:30.444702 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":438,"threads_started":1}
I20260812 07:04:30.445456 15192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:30.434221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:39641 (local address 127.0.0.1:59740)
0812 07:04:30.435065 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:30.435082 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:30.435242 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:30.438242 (+ 3000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:30.438250 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:30.438288 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:30.438668 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:30.438683 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:30.440772 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:30.440779 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:30.442840 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:30.442857 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:30.444683 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:30.444690 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:30.444873 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:30.444912 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 07:04:30.444977 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 07:04:30.445143 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":551,"threads_started":1}
I20260812 07:04:30.446338 15197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:30.436070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59740 (local address 127.6.61.61:39641)
0812 07:04:30.437255 (+ 1185us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:30.437265 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:30.437290 (+ 25us) server_negotiation.cc:407] Connection header received
0812 07:04:30.437381 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:30.437389 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:30.437487 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:30.438835 (+ 1348us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:30.439178 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:30.440574 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:30.443046 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:30.444478 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:30.445450 (+ 972us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:30.445711 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:30.445770 (+ 59us) server_negotiation.cc:299] Negotiation successful
0812 07:04:30.445943 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":1002,"thread_start_us":437,"threads_started":1}
I20260812 07:04:30.446313 14575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "06a305d836de4590a62f589f803f0dfa" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" is_pre_election: true
I20260812 07:04:30.446697 14575 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06a305d836de4590a62f589f803f0dfa in term 1.
I20260812 07:04:30.447085 14360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "06a305d836de4590a62f589f803f0dfa" is_pre_election: true
I20260812 07:04:30.447203 14329 leader_election.cc:302] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06a305d836de4590a62f589f803f0dfa, b9758e0d5bf44fa68a4085ab311c9f43; no voters: 6c86051a8e4146a3b059183c03351248
I20260812 07:04:30.447450 14360 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9758e0d5bf44fa68a4085ab311c9f43 in term 1.
I20260812 07:04:30.447597 15189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 07:04:30.447727 15189 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6c86051a8e4146a3b059183c03351248)
I20260812 07:04:30.447834 15189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 1 FOLLOWER]: Advancing to term 2
I20260812 07:04:30.448006 14550 leader_election.cc:302] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06a305d836de4590a62f589f803f0dfa, b9758e0d5bf44fa68a4085ab311c9f43; no voters: 6c86051a8e4146a3b059183c03351248
I20260812 07:04:30.448385 15187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 07:04:30.448508 15187 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6c86051a8e4146a3b059183c03351248)
I20260812 07:04:30.448603 15187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [term 1 FOLLOWER]: Advancing to term 2
I20260812 07:04:30.450997 15187 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 34607 } attrs { promote: false } }
I20260812 07:04:30.451085 15189 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 34607 } attrs { promote: false } }
I20260812 07:04:30.451766 15189 leader_election.cc:288] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [CANDIDATE]: Term 2 election: Requested vote from peers 6c86051a8e4146a3b059183c03351248 (127.6.61.62:34449), b9758e0d5bf44fa68a4085ab311c9f43 (127.6.61.60:34607)
I20260812 07:04:30.452196 15192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:30.451781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.0.0.1:37756)
0812 07:04:30.451949 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:30.452046 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260812 07:04:30.452565 15187 leader_election.cc:288] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [CANDIDATE]: Term 2 election: Requested vote from peers 6c86051a8e4146a3b059183c03351248 (127.6.61.62:34449), 06a305d836de4590a62f589f803f0dfa (127.6.61.61:39641)
I20260812 07:04:30.453065 14360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "06a305d836de4590a62f589f803f0dfa"
I20260812 07:04:30.453061 14575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "06a305d836de4590a62f589f803f0dfa" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b9758e0d5bf44fa68a4085ab311c9f43"
I20260812 07:04:30.453346 14360 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b9758e0d5bf44fa68a4085ab311c9f43 in current term 2: Already voted for candidate 06a305d836de4590a62f589f803f0dfa in this term.
I20260812 07:04:30.453384 14575 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 06a305d836de4590a62f589f803f0dfa in current term 2: Already voted for candidate b9758e0d5bf44fa68a4085ab311c9f43 in this term.
I20260812 07:04:30.453452 15192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:30.453079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.0.0.1:37766)
0812 07:04:30.453246 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:30.453331 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260812 07:04:30.453627 14548 leader_election.cc:334] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6c86051a8e4146a3b059183c03351248 (127.6.61.62:34449): Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
I20260812 07:04:30.453961 15194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:30.452144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.0.0.1:37760)
0812 07:04:30.452829 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:30.453749 (+ 920us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":580}
I20260812 07:04:30.454141 14550 leader_election.cc:302] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9758e0d5bf44fa68a4085ab311c9f43; no voters: 06a305d836de4590a62f589f803f0dfa, 6c86051a8e4146a3b059183c03351248
I20260812 07:04:30.454427 15187 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 07:04:30.454957 15194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:30.454637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.0.0.1:37772)
0812 07:04:30.454800 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:30.454865 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 07:04:30.455176 14329 leader_election.cc:334] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6c86051a8e4146a3b059183c03351248 (127.6.61.62:34449): Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
I20260812 07:04:30.455415 14329 leader_election.cc:302] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06a305d836de4590a62f589f803f0dfa; no voters: 6c86051a8e4146a3b059183c03351248, b9758e0d5bf44fa68a4085ab311c9f43
I20260812 07:04:30.455715 15189 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 07:04:30.913201 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:30.912216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.6.61.2:45391)
0812 07:04:30.912772 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:30.912890 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":307,"threads_started":1}
I20260812 07:04:30.918680 15202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:30.917510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.6.61.1:48011)
0812 07:04:30.918310 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:30.918409 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":382,"threads_started":1}
I20260812 07:04:30.921563 15203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:30.920754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.6.61.3:49313)
0812 07:04:30.921248 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:30.921376 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":286,"threads_started":1}
I20260812 07:04:31.915020 15204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:31.914251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.6.61.2:40581)
0812 07:04:31.914750 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:31.914837 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":240,"threads_started":1}
I20260812 07:04:31.920645 15205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:31.919659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.6.61.1:37887)
0812 07:04:31.920302 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:31.920385 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":332,"threads_started":1}
I20260812 07:04:31.923318 15206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:31.922464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.6.61.3:42083)
0812 07:04:31.922969 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:31.923085 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":278,"threads_started":1}
I20260812 07:04:32.318488 15207 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:04:32.318723 15207 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 34607 } attrs { promote: false } }
I20260812 07:04:32.319325 15207 leader_election.cc:288] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6c86051a8e4146a3b059183c03351248 (127.6.61.62:34449), b9758e0d5bf44fa68a4085ab311c9f43 (127.6.61.60:34607)
I20260812 07:04:32.320559 15208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:32.319680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.0.0.1:38018)
0812 07:04:32.320247 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:32.320371 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":273,"threads_started":1}
I20260812 07:04:32.320694 14575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "06a305d836de4590a62f589f803f0dfa" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" is_pre_election: true
I20260812 07:04:32.320989 14575 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06a305d836de4590a62f589f803f0dfa in term 2.
I20260812 07:04:32.321803 14329 leader_election.cc:302] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06a305d836de4590a62f589f803f0dfa, b9758e0d5bf44fa68a4085ab311c9f43; no voters:
I20260812 07:04:32.321998 15208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:32.321550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.0.0.1:38028)
0812 07:04:32.321714 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:32.321802 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 07:04:32.322170 14329 leader_election.cc:334] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6c86051a8e4146a3b059183c03351248 (127.6.61.62:34449): Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
I20260812 07:04:32.322190 15207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 07:04:32.322365 15207 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:04:32.322479 15207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 2 FOLLOWER]: Advancing to term 3
I20260812 07:04:32.324566 15207 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 34607 } attrs { promote: false } }
I20260812 07:04:32.325178 15207 leader_election.cc:288] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [CANDIDATE]: Term 3 election: Requested vote from peers 6c86051a8e4146a3b059183c03351248 (127.6.61.62:34449), b9758e0d5bf44fa68a4085ab311c9f43 (127.6.61.60:34607)
I20260812 07:04:32.325922 15208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:32.325410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.0.0.1:38034)
0812 07:04:32.325573 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:32.325698 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260812 07:04:32.326164 14575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "06a305d836de4590a62f589f803f0dfa" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b9758e0d5bf44fa68a4085ab311c9f43"
I20260812 07:04:32.326418 14575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [term 2 FOLLOWER]: Advancing to term 3
I20260812 07:04:32.327253 15208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:32.326896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.0.0.1:38044)
0812 07:04:32.327061 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:32.327139 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260812 07:04:32.327417 14329 leader_election.cc:334] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 6c86051a8e4146a3b059183c03351248 (127.6.61.62:34449): Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
I20260812 07:04:32.328804 14575 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06a305d836de4590a62f589f803f0dfa in term 3.
I20260812 07:04:32.329371 14329 leader_election.cc:302] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06a305d836de4590a62f589f803f0dfa, b9758e0d5bf44fa68a4085ab311c9f43; no voters: 6c86051a8e4146a3b059183c03351248
I20260812 07:04:32.329924 15207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 3 FOLLOWER]: Leader election won for term 3
I20260812 07:04:32.330379 15207 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [term 3 LEADER]: Becoming Leader. State: Replica: 06a305d836de4590a62f589f803f0dfa, State: Running, Role: LEADER
I20260812 07:04:32.330806 15207 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 34607 } attrs { promote: false } }
I20260812 07:04:32.332546 15210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06a305d836de4590a62f589f803f0dfa. Latest consensus state: current_term: 3 leader_uuid: "06a305d836de4590a62f589f803f0dfa" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 34607 } attrs { promote: false } } }
I20260812 07:04:32.332931 15210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:32.333701 15212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:04:32.337608 15212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:04:32.338526 15212 catalog_manager.cc:1259] Loaded cluster ID: 71d35a330bec42eb92f085ce763b979e
I20260812 07:04:32.338630 15212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:04:32.339792 15212 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:04:32.340705 15212 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa: Loaded TSK: 0
I20260812 07:04:32.341580 15212 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:04:32.394649 14349 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:04:32.402127 14349 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:04:32.406477 15208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:32.406042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.0.0.1:38060)
0812 07:04:32.406201 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:32.406328 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260812 07:04:32.406976 14575 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [term 3 FOLLOWER]: Refusing update from remote peer 06a305d836de4590a62f589f803f0dfa: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260812 07:04:32.407625 15208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:32.407231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.0.0.1:38076)
0812 07:04:32.407407 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:32.407479 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260812 07:04:32.407755 15210 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 34607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260812 07:04:32.408109 14329 consensus_peers.cc:487] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa -> Peer 6c86051a8e4146a3b059183c03351248 (127.6.61.62:34449): Couldn't send request to peer 6c86051a8e4146a3b059183c03351248. Status: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 07:04:32.410813 15214 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06a305d836de4590a62f589f803f0dfa. Latest consensus state: current_term: 3 leader_uuid: "06a305d836de4590a62f589f803f0dfa" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 34607 } attrs { promote: false } } }
I20260812 07:04:32.411165 15214 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:04:32.412542 15210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "06a305d836de4590a62f589f803f0dfa" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 34607 } attrs { promote: false } } }
I20260812 07:04:32.412842 15210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:32.417970 15215 mvcc.cc:205] Tried to move back new op lower bound from 7317578843770081280 to 7317578843470503936. Current Snapshot: MvccSnapshot[applied={T|T < 7317578843770081280}]
I20260812 07:04:32.418412 15214 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "06a305d836de4590a62f589f803f0dfa" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c86051a8e4146a3b059183c03351248" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34449 } } peers { permanent_uuid: "06a305d836de4590a62f589f803f0dfa" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 39641 } attrs { promote: false } } peers { permanent_uuid: "b9758e0d5bf44fa68a4085ab311c9f43" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 34607 } attrs { promote: false } } }
I20260812 07:04:32.418696 15214 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9758e0d5bf44fa68a4085ab311c9f43 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:04:32.433619 15222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:32.425088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60584 (local address 127.6.61.1:42623)
0812 07:04:32.425982 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:32.425987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:32.426002 (+ 15us) server_negotiation.cc:407] Connection header received
0812 07:04:32.426105 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:32.426112 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:32.426199 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:32.426858 (+ 659us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:32.426986 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:32.428357 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:32.430285 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:32.432024 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:32.432194 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:32.432347 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:32.433147 (+ 800us) server_negotiation.cc:299] Negotiation successful
0812 07:04:32.433309 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":788,"thread_start_us":677,"threads_started":1}
I20260812 07:04:32.433929 15221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:32.424632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:42623 (local address 127.0.0.1:60584)
0812 07:04:32.425200 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:32.425231 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:32.425534 (+ 303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:32.426377 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:32.426382 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:32.426412 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:32.426704 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:32.426713 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:32.428498 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:32.428504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:32.430033 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:32.430061 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:32.432495 (+ 2434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:32.432505 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:32.432857 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:32.432971 (+ 114us) client_negotiation.cc:769] Sending connection context
0812 07:04:32.433099 (+ 128us) client_negotiation.cc:240] Negotiation successful
0812 07:04:32.433297 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":28,"thread_start_us":305,"threads_started":1}
I20260812 07:04:32.437280 15208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:32.423819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:39417 (local address 127.0.0.1:39386)
0812 07:04:32.424087 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:32.424107 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:32.424261 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:32.430528 (+ 6267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:32.430533 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:32.430558 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:32.430863 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:32.430876 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:32.432447 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:32.432455 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:32.434431 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:32.434445 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:32.436059 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:32.436069 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:32.436349 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:32.436479 (+ 130us) client_negotiation.cc:769] Sending connection context
0812 07:04:32.436592 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 07:04:32.436859 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":45}
I20260812 07:04:32.438006 14702 tablet_service.cc:1467] Processing CreateTablet for tablet cf578312acfd4b8587089a237a8199a7 (DEFAULT_TABLE table=test-table [id=60360db8f8ab4dd5b1eec52f6b93deb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:04:32.438393 15220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:32.424049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39386 (local address 127.6.61.2:39417)
0812 07:04:32.430261 (+ 6212us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:32.430266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:32.430282 (+ 16us) server_negotiation.cc:407] Connection header received
0812 07:04:32.430340 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:32.430343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:32.430398 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:32.430511 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:32.431025 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:32.432318 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:32.434595 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:32.435812 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:32.435944 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:32.436053 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:32.437924 (+ 1871us) server_negotiation.cc:299] Negotiation successful
0812 07:04:32.438105 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6050,"thread_start_us":5932,"threads_started":1}
I20260812 07:04:32.438583 14702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf578312acfd4b8587089a237a8199a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:32.439184 14859 tablet_service.cc:1467] Processing CreateTablet for tablet cf578312acfd4b8587089a237a8199a7 (DEFAULT_TABLE table=test-table [id=60360db8f8ab4dd5b1eec52f6b93deb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:04:32.439610 14859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf578312acfd4b8587089a237a8199a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:32.441289 15221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:32.425409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:34313 (local address 127.0.0.1:60332)
0812 07:04:32.434501 (+ 9092us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:32.434527 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:32.434676 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:32.435089 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:32.435096 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:32.435129 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:32.435354 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:32.435373 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:32.437598 (+ 2225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:32.437605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:32.439328 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:32.439342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:32.440529 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:32.440535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:32.440736 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:32.440759 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 07:04:32.440868 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 07:04:32.441005 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8954,"thread_start_us":11672,"threads_started":1}
I20260812 07:04:32.441325 15224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:32.425792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60332 (local address 127.6.61.3:34313)
0812 07:04:32.426382 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:32.426389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:32.434613 (+ 8224us) server_negotiation.cc:407] Connection header received
0812 07:04:32.434834 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:32.434840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:32.434938 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:32.435048 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:32.435525 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:32.436589 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:32.439510 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:32.440360 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:32.440508 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:32.440614 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:32.440898 (+ 284us) server_negotiation.cc:299] Negotiation successful
0812 07:04:32.441039 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":303,"threads_started":1}
I20260812 07:04:32.442090 15028 tablet_service.cc:1467] Processing CreateTablet for tablet cf578312acfd4b8587089a237a8199a7 (DEFAULT_TABLE table=test-table [id=60360db8f8ab4dd5b1eec52f6b93deb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:04:32.442579 15028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf578312acfd4b8587089a237a8199a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:32.446825 15225 tablet_bootstrap.cc:492] T cf578312acfd4b8587089a237a8199a7 P d6f37996431d4b60afeb49f20e3fbba4: Bootstrap starting.
I20260812 07:04:32.448498 15226 tablet_bootstrap.cc:492] T cf578312acfd4b8587089a237a8199a7 P 1a5f51df7c0c43bc85366d1c15dc2f89: Bootstrap starting.
I20260812 07:04:32.448895 15225 tablet_bootstrap.cc:654] T cf578312acfd4b8587089a237a8199a7 P d6f37996431d4b60afeb49f20e3fbba4: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:32.450304 15226 tablet_bootstrap.cc:654] T cf578312acfd4b8587089a237a8199a7 P 1a5f51df7c0c43bc85366d1c15dc2f89: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:32.452749 15226 tablet_bootstrap.cc:492] T cf578312acfd4b8587089a237a8199a7 P 1a5f51df7c0c43bc85366d1c15dc2f89: No bootstrap required, opened a new log
I20260812 07:04:32.453001 15226 ts_tablet_manager.cc:1397] T cf578312acfd4b8587089a237a8199a7 P 1a5f51df7c0c43bc85366d1c15dc2f89: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 07:04:32.453070 15229 tablet_bootstrap.cc:492] T cf578312acfd4b8587089a237a8199a7 P 30fd1f2ae015406199f6b131c77b167f: Bootstrap starting.
I20260812 07:04:32.453871 15226 raft_consensus.cc:348] T cf578312acfd4b8587089a237a8199a7 P 1a5f51df7c0c43bc85366d1c15dc2f89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f37996431d4b60afeb49f20e3fbba4" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 39417 } } peers { permanent_uuid: "30fd1f2ae015406199f6b131c77b167f" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42623 } } peers { permanent_uuid: "1a5f51df7c0c43bc85366d1c15dc2f89" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 34313 } }
I20260812 07:04:32.454214 15226 raft_consensus.cc:374] T cf578312acfd4b8587089a237a8199a7 P 1a5f51df7c0c43bc85366d1c15dc2f89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:32.454347 15226 raft_consensus.cc:729] T cf578312acfd4b8587089a237a8199a7 P 1a5f51df7c0c43bc85366d1c15dc2f89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a5f51df7c0c43bc85366d1c15dc2f89, State: Initialized, Role: FOLLOWER
I20260812 07:04:32.454631 15226 consensus_queue.cc:260] T cf578312acfd4b8587089a237a8199a7 P 1a5f51df7c0c43bc85366d1c15dc2f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f37996431d4b60afeb49f20e3fbba4" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 39417 } } peers { permanent_uuid: "30fd1f2ae015406199f6b131c77b167f" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42623 } } peers { permanent_uuid: "1a5f51df7c0c43bc85366d1c15dc2f89" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 34313 } }
I20260812 07:04:32.455514 15229 tablet_bootstrap.cc:654] T cf578312acfd4b8587089a237a8199a7 P 30fd1f2ae015406199f6b131c77b167f: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:32.461565 15226 ts_tablet_manager.cc:1428] T cf578312acfd4b8587089a237a8199a7 P 1a5f51df7c0c43bc85366d1c15dc2f89: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260812 07:04:32.466533 15232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:32.463114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.6.61.3:52739)
0812 07:04:32.465995 (+ 2881us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:32.466176 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2669,"thread_start_us":2493,"threads_started":1}
I20260812 07:04:32.468545 15225 tablet_bootstrap.cc:492] T cf578312acfd4b8587089a237a8199a7 P d6f37996431d4b60afeb49f20e3fbba4: No bootstrap required, opened a new log
I20260812 07:04:32.468722 15225 ts_tablet_manager.cc:1397] T cf578312acfd4b8587089a237a8199a7 P d6f37996431d4b60afeb49f20e3fbba4: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.013s
I20260812 07:04:32.469359 15102 heartbeater.cc:503] Master 127.6.61.61:39641 was elected leader, sending a full tablet report...
I20260812 07:04:32.469662 15225 raft_consensus.cc:348] T cf578312acfd4b8587089a237a8199a7 P d6f37996431d4b60afeb49f20e3fbba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f37996431d4b60afeb49f20e3fbba4" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 39417 } } peers { permanent_uuid: "30fd1f2ae015406199f6b131c77b167f" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42623 } } peers { permanent_uuid: "1a5f51df7c0c43bc85366d1c15dc2f89" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 34313 } }
I20260812 07:04:32.470136 15225 raft_consensus.cc:374] T cf578312acfd4b8587089a237a8199a7 P d6f37996431d4b60afeb49f20e3fbba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:32.470314 15225 raft_consensus.cc:729] T cf578312acfd4b8587089a237a8199a7 P d6f37996431d4b60afeb49f20e3fbba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6f37996431d4b60afeb49f20e3fbba4, State: Initialized, Role: FOLLOWER
I20260812 07:04:32.470688 15225 consensus_queue.cc:260] T cf578312acfd4b8587089a237a8199a7 P d6f37996431d4b60afeb49f20e3fbba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f37996431d4b60afeb49f20e3fbba4" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 39417 } } peers { permanent_uuid: "30fd1f2ae015406199f6b131c77b167f" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42623 } } peers { permanent_uuid: "1a5f51df7c0c43bc85366d1c15dc2f89" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 34313 } }
I20260812 07:04:32.471289 15229 tablet_bootstrap.cc:492] T cf578312acfd4b8587089a237a8199a7 P 30fd1f2ae015406199f6b131c77b167f: No bootstrap required, opened a new log
I20260812 07:04:32.471503 15229 ts_tablet_manager.cc:1397] T cf578312acfd4b8587089a237a8199a7 P 30fd1f2ae015406199f6b131c77b167f: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.004s
I20260812 07:04:32.471676 15232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:32.467659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.6.61.3:32923)
0812 07:04:32.471421 (+ 3762us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:32.471524 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3634}
I20260812 07:04:32.472569 15229 raft_consensus.cc:348] T cf578312acfd4b8587089a237a8199a7 P 30fd1f2ae015406199f6b131c77b167f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f37996431d4b60afeb49f20e3fbba4" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 39417 } } peers { permanent_uuid: "30fd1f2ae015406199f6b131c77b167f" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42623 } } peers { permanent_uuid: "1a5f51df7c0c43bc85366d1c15dc2f89" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 34313 } }
I20260812 07:04:32.472921 15229 raft_consensus.cc:374] T cf578312acfd4b8587089a237a8199a7 P 30fd1f2ae015406199f6b131c77b167f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:32.473038 15229 raft_consensus.cc:729] T cf578312acfd4b8587089a237a8199a7 P 30fd1f2ae015406199f6b131c77b167f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30fd1f2ae015406199f6b131c77b167f, State: Initialized, Role: FOLLOWER
I20260812 07:04:32.473356 15229 consensus_queue.cc:260] T cf578312acfd4b8587089a237a8199a7 P 30fd1f2ae015406199f6b131c77b167f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f37996431d4b60afeb49f20e3fbba4" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 39417 } } peers { permanent_uuid: "30fd1f2ae015406199f6b131c77b167f" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42623 } } peers { permanent_uuid: "1a5f51df7c0c43bc85366d1c15dc2f89" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 34313 } }
I20260812 07:04:32.474408 15225 ts_tablet_manager.cc:1428] T cf578312acfd4b8587089a237a8199a7 P d6f37996431d4b60afeb49f20e3fbba4: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260812 07:04:32.474442 15229 ts_tablet_manager.cc:1428] T cf578312acfd4b8587089a237a8199a7 P 30fd1f2ae015406199f6b131c77b167f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260812 07:04:32.479355 15236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:32.475217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.6.61.1:49021)
0812 07:04:32.478001 (+ 2784us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:32.478209 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2631,"thread_start_us":2430,"threads_started":1}
I20260812 07:04:32.481421 14773 heartbeater.cc:503] Master 127.6.61.61:39641 was elected leader, sending a full tablet report...
I20260812 07:04:32.482204 15236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:32.481733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.6.61.1:47633)
0812 07:04:32.481941 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:32.482036 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260812 07:04:32.486555 15235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:32.475150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.6.61.2:54803)
0812 07:04:32.486190 (+ 11040us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:32.486321 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10928,"threads_started":1}
I20260812 07:04:32.487772 15235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:32.487397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.6.61.2:52663)
0812 07:04:32.487598 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:32.487664 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260812 07:04:32.500294 15233 raft_consensus.cc:482] T cf578312acfd4b8587089a237a8199a7 P d6f37996431d4b60afeb49f20e3fbba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:04:32.500532 15233 raft_consensus.cc:504] T cf578312acfd4b8587089a237a8199a7 P d6f37996431d4b60afeb49f20e3fbba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f37996431d4b60afeb49f20e3fbba4" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 39417 } } peers { permanent_uuid: "30fd1f2ae015406199f6b131c77b167f" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42623 } } peers { permanent_uuid: "1a5f51df7c0c43bc85366d1c15dc2f89" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 34313 } }
I20260812 07:04:32.501250 15233 leader_election.cc:288] T cf578312acfd4b8587089a237a8199a7 P d6f37996431d4b60afeb49f20e3fbba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30fd1f2ae015406199f6b131c77b167f (127.6.61.1:42623), 1a5f51df7c0c43bc85366d1c15dc2f89 (127.6.61.3:34313)
I20260812 07:04:32.502018 14724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf578312acfd4b8587089a237a8199a7" candidate_uuid: "d6f37996431d4b60afeb49f20e3fbba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30fd1f2ae015406199f6b131c77b167f" is_pre_election: true
I20260812 07:04:32.502151 15050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf578312acfd4b8587089a237a8199a7" candidate_uuid: "d6f37996431d4b60afeb49f20e3fbba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a5f51df7c0c43bc85366d1c15dc2f89" is_pre_election: true
I20260812 07:04:32.502287 14724 raft_consensus.cc:2454] T cf578312acfd4b8587089a237a8199a7 P 30fd1f2ae015406199f6b131c77b167f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6f37996431d4b60afeb49f20e3fbba4 in term 0.
I20260812 07:04:32.502471 15050 raft_consensus.cc:2454] T cf578312acfd4b8587089a237a8199a7 P 1a5f51df7c0c43bc85366d1c15dc2f89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6f37996431d4b60afeb49f20e3fbba4 in term 0.
I20260812 07:04:32.502816 14807 leader_election.cc:302] T cf578312acfd4b8587089a237a8199a7 P d6f37996431d4b60afeb49f20e3fbba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30fd1f2ae015406199f6b131c77b167f, d6f37996431d4b60afeb49f20e3fbba4; no voters:
I20260812 07:04:32.503072 15233 raft_consensus.cc:2781] T cf578312acfd4b8587089a237a8199a7 P d6f37996431d4b60afeb49f20e3fbba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:04:32.503207 15233 raft_consensus.cc:482] T cf578312acfd4b8587089a237a8199a7 P d6f37996431d4b60afeb49f20e3fbba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:04:32.503309 15233 raft_consensus.cc:3037] T cf578312acfd4b8587089a237a8199a7 P d6f37996431d4b60afeb49f20e3fbba4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:32.505370 15233 raft_consensus.cc:504] T cf578312acfd4b8587089a237a8199a7 P d6f37996431d4b60afeb49f20e3fbba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f37996431d4b60afeb49f20e3fbba4" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 39417 } } peers { permanent_uuid: "30fd1f2ae015406199f6b131c77b167f" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42623 } } peers { permanent_uuid: "1a5f51df7c0c43bc85366d1c15dc2f89" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 34313 } }
I20260812 07:04:32.505923 15233 leader_election.cc:288] T cf578312acfd4b8587089a237a8199a7 P d6f37996431d4b60afeb49f20e3fbba4 [CANDIDATE]: Term 1 election: Requested vote from peers 30fd1f2ae015406199f6b131c77b167f (127.6.61.1:42623), 1a5f51df7c0c43bc85366d1c15dc2f89 (127.6.61.3:34313)
I20260812 07:04:32.506451 14724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf578312acfd4b8587089a237a8199a7" candidate_uuid: "d6f37996431d4b60afeb49f20e3fbba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30fd1f2ae015406199f6b131c77b167f"
I20260812 07:04:32.506510 15050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf578312acfd4b8587089a237a8199a7" candidate_uuid: "d6f37996431d4b60afeb49f20e3fbba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a5f51df7c0c43bc85366d1c15dc2f89"
I20260812 07:04:32.506611 14724 raft_consensus.cc:3037] T cf578312acfd4b8587089a237a8199a7 P 30fd1f2ae015406199f6b131c77b167f [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:32.506776 15050 raft_consensus.cc:3037] T cf578312acfd4b8587089a237a8199a7 P 1a5f51df7c0c43bc85366d1c15dc2f89 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:32.508805 14724 raft_consensus.cc:2454] T cf578312acfd4b8587089a237a8199a7 P 30fd1f2ae015406199f6b131c77b167f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6f37996431d4b60afeb49f20e3fbba4 in term 1.
I20260812 07:04:32.508960 15050 raft_consensus.cc:2454] T cf578312acfd4b8587089a237a8199a7 P 1a5f51df7c0c43bc85366d1c15dc2f89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6f37996431d4b60afeb49f20e3fbba4 in term 1.
I20260812 07:04:32.509263 14807 leader_election.cc:302] T cf578312acfd4b8587089a237a8199a7 P d6f37996431d4b60afeb49f20e3fbba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30fd1f2ae015406199f6b131c77b167f, d6f37996431d4b60afeb49f20e3fbba4; no voters:
I20260812 07:04:32.509577 15233 raft_consensus.cc:2781] T cf578312acfd4b8587089a237a8199a7 P d6f37996431d4b60afeb49f20e3fbba4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:04:32.510036 15233 raft_consensus.cc:686] T cf578312acfd4b8587089a237a8199a7 P d6f37996431d4b60afeb49f20e3fbba4 [term 1 LEADER]: Becoming Leader. State: Replica: d6f37996431d4b60afeb49f20e3fbba4, State: Running, Role: LEADER
I20260812 07:04:32.510407 14943 heartbeater.cc:503] Master 127.6.61.61:39641 was elected leader, sending a full tablet report...
I20260812 07:04:32.510361 15233 consensus_queue.cc:237] T cf578312acfd4b8587089a237a8199a7 P d6f37996431d4b60afeb49f20e3fbba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f37996431d4b60afeb49f20e3fbba4" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 39417 } } peers { permanent_uuid: "30fd1f2ae015406199f6b131c77b167f" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42623 } } peers { permanent_uuid: "1a5f51df7c0c43bc85366d1c15dc2f89" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 34313 } }
I20260812 07:04:32.511171 15235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:32.510734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.6.61.2:43433)
0812 07:04:32.510926 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:32.511004 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260812 07:04:32.513441 14350 catalog_manager.cc:5495] T cf578312acfd4b8587089a237a8199a7 P d6f37996431d4b60afeb49f20e3fbba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6f37996431d4b60afeb49f20e3fbba4 (127.6.61.2). New cstate: current_term: 1 leader_uuid: "d6f37996431d4b60afeb49f20e3fbba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f37996431d4b60afeb49f20e3fbba4" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30fd1f2ae015406199f6b131c77b167f" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a5f51df7c0c43bc85366d1c15dc2f89" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 34313 } health_report { overall_health: UNKNOWN } } }
I20260812 07:04:32.556996 15220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:32.551348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39398 (local address 127.6.61.2:39417)
0812 07:04:32.551560 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:32.551566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:32.551649 (+ 83us) server_negotiation.cc:407] Connection header received
0812 07:04:32.551900 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:32.551904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:32.551989 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:32.552067 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:04:32.552475 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:32.553162 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:32.555081 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:32.555563 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:32.555664 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:32.556257 (+ 593us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:04:32.556389 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:04:32.556460 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:32.556624 (+ 164us) server_negotiation.cc:299] Negotiation successful
0812 07:04:32.556694 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 07:04:32.560524 14724 raft_consensus.cc:1261] T cf578312acfd4b8587089a237a8199a7 P 30fd1f2ae015406199f6b131c77b167f [term 1 FOLLOWER]: Refusing update from remote peer d6f37996431d4b60afeb49f20e3fbba4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:04:32.560534 15050 raft_consensus.cc:1261] T cf578312acfd4b8587089a237a8199a7 P 1a5f51df7c0c43bc85366d1c15dc2f89 [term 1 FOLLOWER]: Refusing update from remote peer d6f37996431d4b60afeb49f20e3fbba4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:04:32.561257 15233 consensus_queue.cc:1035] T cf578312acfd4b8587089a237a8199a7 P d6f37996431d4b60afeb49f20e3fbba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30fd1f2ae015406199f6b131c77b167f" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:04:32.561626 15237 consensus_queue.cc:1035] T cf578312acfd4b8587089a237a8199a7 P d6f37996431d4b60afeb49f20e3fbba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a5f51df7c0c43bc85366d1c15dc2f89" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:04:32.563675 15236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:32.563175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.6.61.1:48325)
0812 07:04:32.563345 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:32.563470 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260812 07:04:32.565289 15232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:32.564650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.6.61.3:47635)
0812 07:04:32.564835 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:32.564960 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260812 07:04:32.571045 15236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:32.570221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.6.61.1:50419)
0812 07:04:32.570772 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:32.570872 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435}
I20260812 07:04:32.575155 15235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:32.574659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.6.61.2:59781)
0812 07:04:32.574828 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:32.574967 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260812 07:04:32.576733 15232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:32.576262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.6.61.3:54665)
0812 07:04:32.576470 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:32.576565 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260812 07:04:32.577680 15235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:32.577253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.6.61.2:53043)
0812 07:04:32.577409 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:32.577499 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260812 07:04:32.582096 14350 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:04:32.582393 14350 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:04:32.589699 14350 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 06a305d836de4590a62f589f803f0dfa: Sending DeleteTablet for 3 replicas of tablet cf578312acfd4b8587089a237a8199a7
I20260812 07:04:32.590656 14859 tablet_service.cc:1514] Processing DeleteTablet for tablet cf578312acfd4b8587089a237a8199a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:04:32 UTC) from {username='oryx', principal='oryx/127.6.61.61@KRBTEST.COM'} at 127.0.0.1:39386
I20260812 07:04:32.591163 15028 tablet_service.cc:1514] Processing DeleteTablet for tablet cf578312acfd4b8587089a237a8199a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:04:32 UTC) from {username='oryx', principal='oryx/127.6.61.61@KRBTEST.COM'} at 127.0.0.1:60332
I20260812 07:04:32.591179 14702 tablet_service.cc:1514] Processing DeleteTablet for tablet cf578312acfd4b8587089a237a8199a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:04:32 UTC) from {username='oryx', principal='oryx/127.6.61.61@KRBTEST.COM'} at 127.0.0.1:60584
I20260812 07:04:32.591389 15255 tablet_replica.cc:330] T cf578312acfd4b8587089a237a8199a7 P d6f37996431d4b60afeb49f20e3fbba4: stopping tablet replica
I20260812 07:04:32.591574 15255 raft_consensus.cc:2229] T cf578312acfd4b8587089a237a8199a7 P d6f37996431d4b60afeb49f20e3fbba4 [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:04:32.592036 15255 raft_consensus.cc:2258] T cf578312acfd4b8587089a237a8199a7 P d6f37996431d4b60afeb49f20e3fbba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:04:32.592133 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 14542
I20260812 07:04:32.596115 15255 ts_tablet_manager.cc:1905] T cf578312acfd4b8587089a237a8199a7 P d6f37996431d4b60afeb49f20e3fbba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:04:32.601917 15257 tablet_replica.cc:330] T cf578312acfd4b8587089a237a8199a7 P 1a5f51df7c0c43bc85366d1c15dc2f89: stopping tablet replica
I20260812 07:04:32.602357 15257 raft_consensus.cc:2229] T cf578312acfd4b8587089a237a8199a7 P 1a5f51df7c0c43bc85366d1c15dc2f89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:04:32.602854 15257 raft_consensus.cc:2258] T cf578312acfd4b8587089a237a8199a7 P 1a5f51df7c0c43bc85366d1c15dc2f89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:04:32.604367 15257 ts_tablet_manager.cc:1905] T cf578312acfd4b8587089a237a8199a7 P 1a5f51df7c0c43bc85366d1c15dc2f89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:04:32.604557 15258 tablet_replica.cc:330] T cf578312acfd4b8587089a237a8199a7 P 30fd1f2ae015406199f6b131c77b167f: stopping tablet replica
I20260812 07:04:32.604876 15258 raft_consensus.cc:2229] T cf578312acfd4b8587089a237a8199a7 P 30fd1f2ae015406199f6b131c77b167f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:04:32.605248 15258 raft_consensus.cc:2258] T cf578312acfd4b8587089a237a8199a7 P 30fd1f2ae015406199f6b131c77b167f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:04:32.605297 15255 ts_tablet_manager.cc:1918] T cf578312acfd4b8587089a237a8199a7 P d6f37996431d4b60afeb49f20e3fbba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:04:32.606312 15255 log.cc:1196] T cf578312acfd4b8587089a237a8199a7 P d6f37996431d4b60afeb49f20e3fbba4: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-1/wal/wals/cf578312acfd4b8587089a237a8199a7
I20260812 07:04:32.606869 15255 ts_tablet_manager.cc:1939] T cf578312acfd4b8587089a237a8199a7 P d6f37996431d4b60afeb49f20e3fbba4: Deleting consensus metadata
I20260812 07:04:32.606941 15258 ts_tablet_manager.cc:1905] T cf578312acfd4b8587089a237a8199a7 P 30fd1f2ae015406199f6b131c77b167f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:04:32.608244 14332 catalog_manager.cc:4841] TS d6f37996431d4b60afeb49f20e3fbba4 (127.6.61.2:39417): tablet cf578312acfd4b8587089a237a8199a7 (table test-table [id=60360db8f8ab4dd5b1eec52f6b93deb2]) successfully deleted
I20260812 07:04:32.610761 15257 ts_tablet_manager.cc:1918] T cf578312acfd4b8587089a237a8199a7 P 1a5f51df7c0c43bc85366d1c15dc2f89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:04:32.610895 15257 log.cc:1196] T cf578312acfd4b8587089a237a8199a7 P 1a5f51df7c0c43bc85366d1c15dc2f89: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-2/wal/wals/cf578312acfd4b8587089a237a8199a7
I20260812 07:04:32.611326 15257 ts_tablet_manager.cc:1939] T cf578312acfd4b8587089a237a8199a7 P 1a5f51df7c0c43bc85366d1c15dc2f89: Deleting consensus metadata
I20260812 07:04:32.612540 14332 catalog_manager.cc:4841] TS 1a5f51df7c0c43bc85366d1c15dc2f89 (127.6.61.3:34313): tablet cf578312acfd4b8587089a237a8199a7 (table test-table [id=60360db8f8ab4dd5b1eec52f6b93deb2]) successfully deleted
I20260812 07:04:32.614264 15258 ts_tablet_manager.cc:1918] T cf578312acfd4b8587089a237a8199a7 P 30fd1f2ae015406199f6b131c77b167f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:04:32.614427 15258 log.cc:1196] T cf578312acfd4b8587089a237a8199a7 P 30fd1f2ae015406199f6b131c77b167f: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518208686283-6388-0/minicluster-data/ts-0/wal/wals/cf578312acfd4b8587089a237a8199a7
I20260812 07:04:32.614872 15258 ts_tablet_manager.cc:1939] T cf578312acfd4b8587089a237a8199a7 P 30fd1f2ae015406199f6b131c77b167f: Deleting consensus metadata
I20260812 07:04:32.615850 14332 catalog_manager.cc:4841] TS 30fd1f2ae015406199f6b131c77b167f (127.6.61.1:42623): tablet cf578312acfd4b8587089a237a8199a7 (table test-table [id=60360db8f8ab4dd5b1eec52f6b93deb2]) successfully deleted
I20260812 07:04:32.620518 15222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:32.593331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60596 (local address 127.6.61.1:42623)
0812 07:04:32.593666 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:32.593676 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:32.605882 (+ 12206us) server_negotiation.cc:407] Connection header received
0812 07:04:32.606297 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:32.606302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:32.606382 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:32.606492 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:32.608306 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:32.609374 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:32.610711 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:32.611302 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:32.611443 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:32.614380 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:32.614424 (+ 44us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:32.614446 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:32.614488 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:32.617689 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:32.618441 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:32.618456 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:32.618459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:32.618590 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:32.618983 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:32.618988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:32.618991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:32.619358 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:32.619684 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:32.619969 (+ 285us) server_negotiation.cc:299] Negotiation successful
0812 07:04:32.620196 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260812 07:04:32.621229 14766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:32.717713 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 14542
I20260812 07:04:32.741694 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 14786
I20260812 07:04:32.756240 15220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:32.742764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39410 (local address 127.6.61.2:39417)
0812 07:04:32.742948 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:32.742952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:32.742971 (+ 19us) server_negotiation.cc:407] Connection header received
0812 07:04:32.743068 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:32.743072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:32.743181 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:32.743269 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:32.744671 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:32.745483 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:32.747069 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:32.747510 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:32.747614 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:32.750198 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:32.750215 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:32.750234 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:32.750271 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:32.753308 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:32.754171 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:32.754180 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:32.754183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:32.754346 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:32.754742 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:32.754747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:32.754749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:32.755189 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:32.755442 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:32.755760 (+ 318us) server_negotiation.cc:299] Negotiation successful
0812 07:04:32.755949 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 07:04:32.757043 14936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:32.841960 15221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:32.841450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.0.0.1:38082)
0812 07:04:32.841653 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:32.841762 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260812 07:04:32.843093 15221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:32.842704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34449 (local address 127.0.0.1:38084)
0812 07:04:32.842883 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:32.842961 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260812 07:04:32.882582 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 14786
I20260812 07:04:32.911696 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 14955
I20260812 07:04:32.926059 15224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:32.912770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60334 (local address 127.6.61.3:34313)
0812 07:04:32.912984 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:32.912990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:32.913010 (+ 20us) server_negotiation.cc:407] Connection header received
0812 07:04:32.913219 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:32.913225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:32.913341 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:32.913505 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:32.915053 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:32.915789 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:32.917038 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:32.917480 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:32.917627 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:32.920288 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:32.920304 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:32.920322 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:32.920356 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:32.923174 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:32.923873 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:32.923880 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:32.923883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:32.924015 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:32.924438 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:32.924444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:32.924447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:32.924822 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:32.925170 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:32.925454 (+ 284us) server_negotiation.cc:299] Negotiation successful
0812 07:04:32.925694 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 07:04:32.926761 15095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:33.026283 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 14955
I20260812 07:04:33.050802 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 14179
I20260812 07:04:33.052057 14380 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:33.178349 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 14179
W20260812 07:04:33.207563 14550 connection.cc:535] server connection from 127.0.0.1:52158 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 07:04:33.208082 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 14291
I20260812 07:04:33.209338 14596 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:33.329102 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 14291
2026-08-12T07:04:33Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (12494 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:04:33 dist-test-slave-69rg krb5kdc[15268](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:04:33 dist-test-slave-69rg krb5kdc[15268](info): set up 2 sockets
Aug 12 07:04:33 dist-test-slave-69rg krb5kdc[15268](info): commencing operation
krb5kdc: starting...
W20260812 07:04:34.541203 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.005s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:04:34 dist-test-slave-69rg krb5kdc[15268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:04:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:04:34Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.62.keytab.
Entry for principal oryx/127.6.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.62.keytab.
I20260812 07:04:34.814306 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:35059
--webserver_interface=127.6.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.62.keytab
--principal=oryx/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:35059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:35.157557 15284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:35.157958 15284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:35.158143 15284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:35.165490 15284 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:04:35.165637 15284 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 07:04:35.165737 15284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:35.165867 15284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:35.165969 15284 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:04:35.166059 15284 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:04:35.176683 15284 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:35059
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.62.keytab
--principal=oryx/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:35059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:35.178071 15284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:35.179421 15284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:35.192096 15292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:35.192005 15295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:35.192073 15293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:35.192952 15284 server_base.cc:1034] running on GCE node
I20260812 07:04:35.193781 15284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:35.195238 15284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:35.196475 15284 hybrid_clock.cc:648] HybridClock initialized: now 1786518275196404 us; error 31 us; skew 500 ppm
I20260812 07:04:35.196838 15284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:04:35 dist-test-slave-69rg krb5kdc[15268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518275, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:35.209353 15284 init.cc:377] Logged in from keytab as oryx/127.6.61.62@KRBTEST.COM (short username oryx)
I20260812 07:04:35.211163 15284 webserver.cc:460] Webserver started at http://127.6.61.62:37829/ using document root <none> and password file <none>
I20260812 07:04:35.211939 15284 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:35.212082 15284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:35.212450 15284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:35.215723 15284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "fc0633cfe3c74aa6a8a6d4fbe3cec982"
format_stamp: "Formatted at 2026-08-12 07:04:35 on dist-test-slave-69rg"
server_key: "921ea8fd682b3cd7bd4c877b6c00c3b0"
server_key_iv: "3a16b5704a3016ee49377245cc03933e"
server_key_version: "encryptionkey@0"
I20260812 07:04:35.216624 15284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "fc0633cfe3c74aa6a8a6d4fbe3cec982"
format_stamp: "Formatted at 2026-08-12 07:04:35 on dist-test-slave-69rg"
server_key: "921ea8fd682b3cd7bd4c877b6c00c3b0"
server_key_iv: "3a16b5704a3016ee49377245cc03933e"
server_key_version: "encryptionkey@0"
I20260812 07:04:35.222283 15284 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 07:04:35.225958 15304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:35.227269 15284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:04:35.227476 15284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "fc0633cfe3c74aa6a8a6d4fbe3cec982"
format_stamp: "Formatted at 2026-08-12 07:04:35 on dist-test-slave-69rg"
server_key: "921ea8fd682b3cd7bd4c877b6c00c3b0"
server_key_iv: "3a16b5704a3016ee49377245cc03933e"
server_key_version: "encryptionkey@0"
I20260812 07:04:35.227707 15284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:35.246840 15284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:35.251250 15284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:35.251590 15284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:35.274712 15284 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:35059
I20260812 07:04:35.274740 15374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:35059 every 8 connection(s)
I20260812 07:04:35.276201 15284 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:04:35.280220 15375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:35.282130 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 15284
I20260812 07:04:35.282344 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:04:35.283031 6388 external_mini_cluster.cc:1428] Setting key b83482d7420116fd9766ad51462ae99a
I20260812 07:04:35.292682 15375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc0633cfe3c74aa6a8a6d4fbe3cec982: Bootstrap starting.
I20260812 07:04:35.296847 15375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc0633cfe3c74aa6a8a6d4fbe3cec982: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:35.298213 15375 log.cc:824] T 00000000000000000000000000000000 P fc0633cfe3c74aa6a8a6d4fbe3cec982: Log is configured to *not* fsync() on all Append() calls
Aug 12 07:04:35 dist-test-slave-69rg krb5kdc[15268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM
I20260812 07:04:35.302054 15375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc0633cfe3c74aa6a8a6d4fbe3cec982: No bootstrap required, opened a new log
I20260812 07:04:35.308979 15378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:35.285028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50176 (local address 127.6.61.62:35059)
0812 07:04:35.285917 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:35.285930 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:35.285976 (+ 46us) server_negotiation.cc:407] Connection header received
0812 07:04:35.287726 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:35.287751 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:35.288187 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:35.288513 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:35.290437 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:35.291765 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:35.293521 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:35.294293 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:35.294458 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:35.299796 (+ 5338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:35.299812 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:35.299845 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:35.299890 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:35.304046 (+ 4156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:35.304801 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:35.304811 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:35.304829 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:35.305061 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:35.305450 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:35.305455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:35.305459 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:35.306254 (+ 795us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:35.306586 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:35.306972 (+ 386us) server_negotiation.cc:299] Negotiation successful
0812 07:04:35.307431 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":387,"threads_started":1}
I20260812 07:04:35.312788 15375 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc0633cfe3c74aa6a8a6d4fbe3cec982 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0633cfe3c74aa6a8a6d4fbe3cec982" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35059 } }
I20260812 07:04:35.313148 15375 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc0633cfe3c74aa6a8a6d4fbe3cec982 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:35.313222 15375 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc0633cfe3c74aa6a8a6d4fbe3cec982 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc0633cfe3c74aa6a8a6d4fbe3cec982, State: Initialized, Role: FOLLOWER
I20260812 07:04:35.313735 15375 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc0633cfe3c74aa6a8a6d4fbe3cec982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0633cfe3c74aa6a8a6d4fbe3cec982" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35059 } }
I20260812 07:04:35.314029 15375 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc0633cfe3c74aa6a8a6d4fbe3cec982 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:04:35.314142 15375 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc0633cfe3c74aa6a8a6d4fbe3cec982 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:04:35.314273 15375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc0633cfe3c74aa6a8a6d4fbe3cec982 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:35.316695 15375 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc0633cfe3c74aa6a8a6d4fbe3cec982 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0633cfe3c74aa6a8a6d4fbe3cec982" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35059 } }
I20260812 07:04:35.317487 15375 leader_election.cc:302] T 00000000000000000000000000000000 P fc0633cfe3c74aa6a8a6d4fbe3cec982 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc0633cfe3c74aa6a8a6d4fbe3cec982; no voters:
I20260812 07:04:35.318570 15375 leader_election.cc:288] T 00000000000000000000000000000000 P fc0633cfe3c74aa6a8a6d4fbe3cec982 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:04:35.318696 15380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc0633cfe3c74aa6a8a6d4fbe3cec982 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:04:35.319562 15380 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc0633cfe3c74aa6a8a6d4fbe3cec982 [term 1 LEADER]: Becoming Leader. State: Replica: fc0633cfe3c74aa6a8a6d4fbe3cec982, State: Running, Role: LEADER
I20260812 07:04:35.320253 15380 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc0633cfe3c74aa6a8a6d4fbe3cec982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0633cfe3c74aa6a8a6d4fbe3cec982" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35059 } }
I20260812 07:04:35.321156 15375 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc0633cfe3c74aa6a8a6d4fbe3cec982 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:04:35.323945 15382 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc0633cfe3c74aa6a8a6d4fbe3cec982 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc0633cfe3c74aa6a8a6d4fbe3cec982. Latest consensus state: current_term: 1 leader_uuid: "fc0633cfe3c74aa6a8a6d4fbe3cec982" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0633cfe3c74aa6a8a6d4fbe3cec982" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35059 } } }
I20260812 07:04:35.324499 15382 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc0633cfe3c74aa6a8a6d4fbe3cec982 [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:35.323932 15381 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc0633cfe3c74aa6a8a6d4fbe3cec982 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc0633cfe3c74aa6a8a6d4fbe3cec982" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0633cfe3c74aa6a8a6d4fbe3cec982" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35059 } } }
I20260812 07:04:35.327383 15381 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc0633cfe3c74aa6a8a6d4fbe3cec982 [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:35.328214 15388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:04:35.334720 15388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:04:35.344372 15388 catalog_manager.cc:1347] Generated new cluster ID: 8627cad067ea481387f32f03c62a7971
I20260812 07:04:35.344511 15388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:04:35.394116 15388 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:04:35.395797 15388 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:04:35.405248 15388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc0633cfe3c74aa6a8a6d4fbe3cec982: Generated new TSK 0
I20260812 07:04:35.406474 15388 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.1.keytab.
Entry for principal oryx/127.6.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.1.keytab.
I20260812 07:04:35.486723 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.1:0
--local_ip_for_outbound_sockets=127.6.61.1
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:35059
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.1.keytab
--principal=oryx/127.6.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:35.842072 15406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:35.842419 15406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:35.842590 15406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:35.850032 15406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:35.850196 15406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:35.850344 15406 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 07:04:35.850481 15406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.1
I20260812 07:04:35.860581 15406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.1.keytab
--principal=oryx/127.6.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.61.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.61.62:35059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.1
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:35.862012 15406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:35.863456 15406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:35.877427 15411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:35.877532 15413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:35.877741 15415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:35.878611 15406 server_base.cc:1034] running on GCE node
I20260812 07:04:35.879361 15406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:35.880334 15406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:35.881588 15406 hybrid_clock.cc:648] HybridClock initialized: now 1786518275881509 us; error 55 us; skew 500 ppm
I20260812 07:04:35.881989 15406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:04:35 dist-test-slave-69rg krb5kdc[15268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518275, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:35.885408 15406 init.cc:377] Logged in from keytab as oryx/127.6.61.1@KRBTEST.COM (short username oryx)
I20260812 07:04:35.886983 15406 webserver.cc:460] Webserver started at http://127.6.61.1:37501/ using document root <none> and password file <none>
I20260812 07:04:35.887717 15406 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:35.887869 15406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:35.888219 15406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:35.891361 15406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-0/data/instance:
uuid: "2da9547064764ae9bfb18d37ce320d7c"
format_stamp: "Formatted at 2026-08-12 07:04:35 on dist-test-slave-69rg"
server_key: "d34397d13a6200f072023910973dca71"
server_key_iv: "4a628fe98f95f1b201eef3c3b1ffceda"
server_key_version: "encryptionkey@0"
I20260812 07:04:35.892218 15406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance:
uuid: "2da9547064764ae9bfb18d37ce320d7c"
format_stamp: "Formatted at 2026-08-12 07:04:35 on dist-test-slave-69rg"
server_key: "d34397d13a6200f072023910973dca71"
server_key_iv: "4a628fe98f95f1b201eef3c3b1ffceda"
server_key_version: "encryptionkey@0"
I20260812 07:04:35.897619 15406 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 07:04:35.901438 15425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:35.902858 15406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:04:35.903043 15406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-0/wal
uuid: "2da9547064764ae9bfb18d37ce320d7c"
format_stamp: "Formatted at 2026-08-12 07:04:35 on dist-test-slave-69rg"
server_key: "d34397d13a6200f072023910973dca71"
server_key_iv: "4a628fe98f95f1b201eef3c3b1ffceda"
server_key_version: "encryptionkey@0"
I20260812 07:04:35.903290 15406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:35.929158 15406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:35.934136 15406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:35.934445 15406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:35.937358 15406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:35.937443 15406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:35.937655 15406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:35.937736 15406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 07:04:35 dist-test-slave-69rg krb5kdc[15268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518275, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM
I20260812 07:04:35.962339 15434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:35.938350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:54913 (local address 127.6.61.62:35059)
0812 07:04:35.938893 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:35.938897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:35.940115 (+ 1218us) server_negotiation.cc:407] Connection header received
0812 07:04:35.942438 (+ 2323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:35.942443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:35.942560 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:35.942686 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:35.944895 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:35.945949 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:35.948573 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:35.949010 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:35.949152 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:35.954551 (+ 5399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:35.954560 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:35.954567 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:35.954623 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:35.958550 (+ 3927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:35.959524 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:35.959533 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:35.959539 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:35.959736 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:35.960191 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:35.960199 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:35.960205 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:35.960663 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:35.960847 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:35.961523 (+ 676us) server_negotiation.cc:299] Negotiation successful
0812 07:04:35.961861 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":306,"threads_started":1}
I20260812 07:04:35.963516 15435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:35.939081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:35059 (local address 127.6.61.1:54913)
0812 07:04:35.939949 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:35.939994 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:35.942130 (+ 2136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:35.942940 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:35.942948 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:35.943784 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:35.944704 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:35.944727 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:35.946156 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:35.946160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:35.948407 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:35.948420 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:35.949298 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:35.949307 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:35.949672 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:35.950626 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:35.950674 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:35.954247 (+ 3573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:35.958721 (+ 4474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:35.958737 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:35.958751 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:35.959370 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:35.959872 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:35.959875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:35.959877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:35.960075 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:35.960795 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:35.960801 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:35.961128 (+ 327us) client_negotiation.cc:769] Sending connection context
0812 07:04:35.961363 (+ 235us) client_negotiation.cc:240] Negotiation successful
0812 07:04:35.961766 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":386,"threads_started":1}
I20260812 07:04:35.986785 15511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:35.969971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:35059 (local address 127.6.61.1:47777)
0812 07:04:35.970693 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:35.970709 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:35.970817 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:35.971213 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:35.971216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:35.971644 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:35.971925 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:35.971934 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:35.973480 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:35.973488 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:35.975343 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:35.975357 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:35.976818 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:35.976825 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:35.977082 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:35.977947 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:35.977974 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:35.979187 (+ 1213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:35.983583 (+ 4396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:35.983591 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:35.983596 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:35.984136 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:35.984679 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:35.984689 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:35.984695 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:35.984902 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:35.985584 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:35.985588 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:35.985746 (+ 158us) client_negotiation.cc:769] Sending connection context
0812 07:04:35.985928 (+ 182us) client_negotiation.cc:240] Negotiation successful
0812 07:04:35.986373 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":421,"threads_started":1}
I20260812 07:04:35.986824 15434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:35.969998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:47777 (local address 127.6.61.62:35059)
0812 07:04:35.970219 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:35.970228 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:35.970788 (+ 560us) server_negotiation.cc:407] Connection header received
0812 07:04:35.970928 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:35.970936 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:35.971051 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:35.971172 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:35.972065 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:35.973323 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:35.975546 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:35.976349 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:35.976494 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:35.979470 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:35.979477 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:35.979483 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:35.979521 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:35.983423 (+ 3902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:35.984298 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:35.984305 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:35.984308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:35.984474 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:35.985082 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:35.985089 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:35.985094 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:35.985458 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:35.985623 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:35.986078 (+ 455us) server_negotiation.cc:299] Negotiation successful
0812 07:04:35.986328 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 07:04:35.988407 15406 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.1:34611
I20260812 07:04:35.988435 15560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.1:34611 every 8 connection(s)
I20260812 07:04:35.989683 15406 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
I20260812 07:04:35.992081 15561 heartbeater.cc:347] Connected to a master server at 127.6.61.62:35059
I20260812 07:04:35.992278 15561 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:35.992858 15561 heartbeater.cc:511] Master 127.6.61.62:35059 requested a full tablet report, sending...
I20260812 07:04:35.994834 15326 ts_manager.cc:194] Registered new tserver with Master: 2da9547064764ae9bfb18d37ce320d7c (127.6.61.1:34611)
I20260812 07:04:35.996356 15326 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.61.1@KRBTEST.COM'} at 127.6.61.1:54913
I20260812 07:04:35.998606 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 15406
I20260812 07:04:35.998812 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance
I20260812 07:04:35.999632 6388 external_mini_cluster.cc:1428] Setting key f969bdfb10482ada5828133abd17e05b
WARNING: no policy specified for oryx/127.6.61.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.61.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.2.keytab.
Entry for principal oryx/127.6.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.2.keytab.
I20260812 07:04:36.061601 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.2:0
--local_ip_for_outbound_sockets=127.6.61.2
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:35059
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.2.keytab
--principal=oryx/127.6.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:36.444413 15565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:36.444701 15565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:36.444801 15565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:36.452128 15565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:36.452255 15565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:36.452342 15565 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 07:04:36.452425 15565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.2
I20260812 07:04:36.463119 15565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.2.keytab
--principal=oryx/127.6.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.61.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.61.62:35059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.2
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:36.464511 15565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:36.465929 15565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:36.479868 15570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:36.479800 15571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:36.479830 15574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:36.481334 15565 server_base.cc:1034] running on GCE node
I20260812 07:04:36.482108 15565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:36.483101 15565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:36.484395 15565 hybrid_clock.cc:648] HybridClock initialized: now 1786518276484306 us; error 62 us; skew 500 ppm
I20260812 07:04:36.484870 15565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:04:36 dist-test-slave-69rg krb5kdc[15268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518276, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:36.488337 15565 init.cc:377] Logged in from keytab as oryx/127.6.61.2@KRBTEST.COM (short username oryx)
I20260812 07:04:36.489885 15565 webserver.cc:460] Webserver started at http://127.6.61.2:36517/ using document root <none> and password file <none>
I20260812 07:04:36.490592 15565 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:36.490684 15565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:36.490978 15565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:36.494132 15565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-1/data/instance:
uuid: "e32364b73e254343b4bb6991207c2f85"
format_stamp: "Formatted at 2026-08-12 07:04:36 on dist-test-slave-69rg"
server_key: "23946bcd1c084736371c958fea362cb6"
server_key_iv: "05993ef76656adc4bbd863e4867dcddb"
server_key_version: "encryptionkey@0"
I20260812 07:04:36.494921 15565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance:
uuid: "e32364b73e254343b4bb6991207c2f85"
format_stamp: "Formatted at 2026-08-12 07:04:36 on dist-test-slave-69rg"
server_key: "23946bcd1c084736371c958fea362cb6"
server_key_iv: "05993ef76656adc4bbd863e4867dcddb"
server_key_version: "encryptionkey@0"
I20260812 07:04:36.500252 15565 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 07:04:36.503767 15581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:36.505111 15565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 07:04:36.505299 15565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-1/wal
uuid: "e32364b73e254343b4bb6991207c2f85"
format_stamp: "Formatted at 2026-08-12 07:04:36 on dist-test-slave-69rg"
server_key: "23946bcd1c084736371c958fea362cb6"
server_key_iv: "05993ef76656adc4bbd863e4867dcddb"
server_key_version: "encryptionkey@0"
I20260812 07:04:36.505534 15565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:36.516760 15565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:36.521392 15565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:36.521731 15565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:36.524610 15565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:36.524727 15565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:36.524874 15565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:36.524956 15565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 07:04:36 dist-test-slave-69rg krb5kdc[15268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518276, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM
I20260812 07:04:36.549845 15590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:36.525367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:41001 (local address 127.6.61.62:35059)
0812 07:04:36.525859 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:36.525868 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:36.527521 (+ 1653us) server_negotiation.cc:407] Connection header received
0812 07:04:36.530719 (+ 3198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:36.530736 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:36.530832 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:36.530969 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:36.533527 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:36.534539 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:36.536552 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:36.537529 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:36.537703 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:36.542592 (+ 4889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:36.542597 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:36.542600 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:36.542635 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:36.546197 (+ 3562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:36.547240 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:36.547248 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:36.547254 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:36.547458 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:36.547988 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:36.547994 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:36.547997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:36.548302 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:36.548473 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:36.549193 (+ 720us) server_negotiation.cc:299] Negotiation successful
0812 07:04:36.549435 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":1286,"threads_started":1}
I20260812 07:04:36.551064 15591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:36.526351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:35059 (local address 127.6.61.2:41001)
0812 07:04:36.527309 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:36.527380 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:36.530296 (+ 2916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:36.531294 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:36.531308 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:36.532283 (+ 975us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:36.533304 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:36.533345 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:36.534727 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:36.534735 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:36.536387 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:36.536401 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:36.537763 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:36.537768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:36.538159 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:36.539033 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:36.539070 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:36.542343 (+ 3273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:36.546348 (+ 4005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:36.546358 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:36.546378 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:36.547070 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:36.547600 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:36.547606 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:36.547611 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:36.547849 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:36.548446 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:36.548455 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:36.548817 (+ 362us) client_negotiation.cc:769] Sending connection context
0812 07:04:36.549092 (+ 275us) client_negotiation.cc:240] Negotiation successful
0812 07:04:36.549628 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":426,"threads_started":1}
I20260812 07:04:36.572887 15642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:36.556801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:35059 (local address 127.6.61.2:58597)
0812 07:04:36.557632 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:36.557648 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:36.557803 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:36.558196 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:36.558201 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:36.558790 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:36.559267 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:36.559279 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:36.561017 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:36.561023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:36.562156 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:36.562168 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:36.563363 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:36.563367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:36.563669 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:36.564454 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:36.564485 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:36.565631 (+ 1146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:36.569617 (+ 3986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:36.569622 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:36.569627 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:36.570147 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:36.570661 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:36.570666 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:36.570670 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:36.570877 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:36.571711 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:36.571719 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:36.571934 (+ 215us) client_negotiation.cc:769] Sending connection context
0812 07:04:36.572103 (+ 169us) client_negotiation.cc:240] Negotiation successful
0812 07:04:36.572534 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":516,"threads_started":1}
I20260812 07:04:36.573877 15590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:36.556765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:58597 (local address 127.6.61.62:35059)
0812 07:04:36.556974 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:36.556982 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:36.557730 (+ 748us) server_negotiation.cc:407] Connection header received
0812 07:04:36.557958 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:36.557963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:36.558045 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:36.558164 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:36.559555 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:36.560858 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:36.562395 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:36.563019 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:36.563154 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:36.565857 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:36.565863 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:36.565869 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:36.565908 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:36.569478 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:36.570356 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:36.570363 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:36.570368 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:36.570530 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:36.571058 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:36.571064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:36.571070 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:36.571541 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:36.573325 (+ 1784us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:36.573448 (+ 123us) server_negotiation.cc:299] Negotiation successful
0812 07:04:36.573620 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 07:04:36.581981 15565 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.2:35663
I20260812 07:04:36.582003 15715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.2:35663 every 8 connection(s)
I20260812 07:04:36.583375 15565 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
I20260812 07:04:36.585554 15716 heartbeater.cc:347] Connected to a master server at 127.6.61.62:35059
I20260812 07:04:36.585856 15716 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:36.586766 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 15565
I20260812 07:04:36.586751 15716 heartbeater.cc:511] Master 127.6.61.62:35059 requested a full tablet report, sending...
I20260812 07:04:36.586944 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance
I20260812 07:04:36.587396 6388 external_mini_cluster.cc:1428] Setting key 09be41e736226d1c1d36bfa5c01c069c
I20260812 07:04:36.588204 15326 ts_manager.cc:194] Registered new tserver with Master: e32364b73e254343b4bb6991207c2f85 (127.6.61.2:35663)
I20260812 07:04:36.589500 15326 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.61.2@KRBTEST.COM'} at 127.6.61.2:41001
WARNING: no policy specified for oryx/127.6.61.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.61.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.3.keytab.
Entry for principal oryx/127.6.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.3.keytab.
I20260812 07:04:36.647418 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.3:0
--local_ip_for_outbound_sockets=127.6.61.3
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:35059
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.3.keytab
--principal=oryx/127.6.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:36.995445 15720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:36.995703 15720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:36.995867 15720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260812 07:04:36.999114 15561 heartbeater.cc:503] Master 127.6.61.62:35059 was elected leader, sending a full tablet report...
W20260812 07:04:37.004071 15720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:37.004181 15720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:37.004295 15720 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 07:04:37.004395 15720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.3
I20260812 07:04:37.014789 15720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.3.keytab
--principal=oryx/127.6.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.61.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.61.62:35059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.3
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:37.016047 15720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:37.017361 15720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:37.030977 15727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:37.030934 15728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:37.030931 15731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:37.032456 15720 server_base.cc:1034] running on GCE node
I20260812 07:04:37.033115 15720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:37.034060 15720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:37.035327 15720 hybrid_clock.cc:648] HybridClock initialized: now 1786518277035238 us; error 61 us; skew 500 ppm
I20260812 07:04:37.035650 15720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:04:37 dist-test-slave-69rg krb5kdc[15268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518277, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:37.039011 15720 init.cc:377] Logged in from keytab as oryx/127.6.61.3@KRBTEST.COM (short username oryx)
I20260812 07:04:37.040544 15720 webserver.cc:460] Webserver started at http://127.6.61.3:34745/ using document root <none> and password file <none>
I20260812 07:04:37.041322 15720 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:37.041419 15720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:37.041767 15720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:37.044986 15720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-2/data/instance:
uuid: "59b3abeeb4124526b938e9dc4cf9a7c0"
format_stamp: "Formatted at 2026-08-12 07:04:37 on dist-test-slave-69rg"
server_key: "8e7568604683e4bcb30d1f0d56242336"
server_key_iv: "13dbde5d9c61227b40f73b17f53b182f"
server_key_version: "encryptionkey@0"
I20260812 07:04:37.045925 15720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance:
uuid: "59b3abeeb4124526b938e9dc4cf9a7c0"
format_stamp: "Formatted at 2026-08-12 07:04:37 on dist-test-slave-69rg"
server_key: "8e7568604683e4bcb30d1f0d56242336"
server_key_iv: "13dbde5d9c61227b40f73b17f53b182f"
server_key_version: "encryptionkey@0"
I20260812 07:04:37.051412 15720 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 07:04:37.055253 15738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:37.056676 15720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:04:37.056874 15720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-2/wal
uuid: "59b3abeeb4124526b938e9dc4cf9a7c0"
format_stamp: "Formatted at 2026-08-12 07:04:37 on dist-test-slave-69rg"
server_key: "8e7568604683e4bcb30d1f0d56242336"
server_key_iv: "13dbde5d9c61227b40f73b17f53b182f"
server_key_version: "encryptionkey@0"
I20260812 07:04:37.057122 15720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:37.082455 15720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:37.086994 15720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:37.087319 15720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:37.090265 15720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:37.090430 15720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:37.090590 15720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:37.090687 15720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 07:04:37 dist-test-slave-69rg krb5kdc[15268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518277, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM
I20260812 07:04:37.116746 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:37.091125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:44891 (local address 127.6.61.62:35059)
0812 07:04:37.094247 (+ 3122us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:37.094256 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:37.094282 (+ 26us) server_negotiation.cc:407] Connection header received
0812 07:04:37.095561 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:37.095567 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:37.095672 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:37.095818 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:37.098406 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.099521 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.102263 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.102943 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.103119 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:37.109032 (+ 5913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:37.109043 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:37.109052 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:37.109111 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:37.113312 (+ 4201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:37.114219 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:37.114231 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:37.114235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:37.114368 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:37.114847 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:37.114851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:37.114853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:37.115122 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:37.115284 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:37.116076 (+ 792us) server_negotiation.cc:299] Negotiation successful
0812 07:04:37.116313 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2978,"thread_start_us":2874,"threads_started":1}
I20260812 07:04:37.118022 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:37.091719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:35059 (local address 127.6.61.3:44891)
0812 07:04:37.092667 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:37.092718 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:37.095002 (+ 2284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:37.096088 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:37.096100 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:37.096999 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:37.098179 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:37.098220 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.099691 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.099707 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:37.102065 (+ 2358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:37.102086 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.103243 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.103252 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:37.103487 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:37.104389 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:37.104435 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:37.108679 (+ 4244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:37.113467 (+ 4788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:37.113477 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:37.113500 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:37.114070 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:37.114491 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:37.114496 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:37.114504 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:37.114712 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:37.115268 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:37.115276 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:37.115654 (+ 378us) client_negotiation.cc:769] Sending connection context
0812 07:04:37.115941 (+ 287us) client_negotiation.cc:240] Negotiation successful
0812 07:04:37.116514 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":413,"threads_started":1}
I20260812 07:04:37.143819 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:37.128096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:46773 (local address 127.6.61.62:35059)
0812 07:04:37.128321 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:37.128329 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:37.129301 (+ 972us) server_negotiation.cc:407] Connection header received
0812 07:04:37.129519 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:37.129524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:37.129593 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:37.129702 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:37.131347 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.132500 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.134839 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.135303 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.135442 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:37.137349 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:37.137354 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:37.137358 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:37.137389 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:37.140776 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:37.141521 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:37.141527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:37.141531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:37.141690 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:37.142250 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:37.142256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:37.142260 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:37.142587 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:37.142759 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:37.143184 (+ 425us) server_negotiation.cc:299] Negotiation successful
0812 07:04:37.143430 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 07:04:37.143893 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:37.128198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:35059 (local address 127.6.61.3:46773)
0812 07:04:37.129162 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:37.129190 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:37.129372 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:37.129762 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:37.129767 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:37.130592 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:37.131140 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:37.131162 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.132743 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.132751 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:37.134670 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:37.134686 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.135514 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.135522 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:37.135683 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:37.136356 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:37.136373 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:37.137132 (+ 759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:37.140920 (+ 3788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:37.140925 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:37.140929 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:37.141375 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:37.141846 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:37.141855 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:37.141862 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:37.142074 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:37.142717 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:37.142722 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:37.142918 (+ 196us) client_negotiation.cc:769] Sending connection context
0812 07:04:37.143061 (+ 143us) client_negotiation.cc:240] Negotiation successful
0812 07:04:37.143441 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":550,"threads_started":1}
I20260812 07:04:37.151271 15720 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.3:33203
I20260812 07:04:37.151305 15881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.3:33203 every 8 connection(s)
I20260812 07:04:37.152520 15720 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
I20260812 07:04:37.154796 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 15720
I20260812 07:04:37.154840 15882 heartbeater.cc:347] Connected to a master server at 127.6.61.62:35059
I20260812 07:04:37.154965 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance
I20260812 07:04:37.155117 15882 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:37.155479 6388 external_mini_cluster.cc:1428] Setting key a45f424a6ca9ce96992735277c0e091c
I20260812 07:04:37.155818 15882 heartbeater.cc:511] Master 127.6.61.62:35059 requested a full tablet report, sending...
I20260812 07:04:37.157388 15326 ts_manager.cc:194] Registered new tserver with Master: 59b3abeeb4124526b938e9dc4cf9a7c0 (127.6.61.3:33203)
I20260812 07:04:37.158349 15326 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.61.3@KRBTEST.COM'} at 127.6.61.3:44891
I20260812 07:04:37.158798 6388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 07:04:37 dist-test-slave-69rg krb5kdc[15268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518277, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 07:04:37 dist-test-slave-69rg krb5kdc[15268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:37 dist-test-slave-69rg krb5kdc[15268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 07:04:37.218740 15747 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:37.207548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50180 (local address 127.6.61.62:35059)
0812 07:04:37.207762 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:37.207781 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:37.208039 (+ 258us) server_negotiation.cc:407] Connection header received
0812 07:04:37.208264 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:37.208269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:37.208357 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:37.208490 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:37.210035 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.210819 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.212100 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.212593 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.212690 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:37.218278 (+ 5588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50180: BlockingRecv error: recv got EOF from 127.0.0.1:50180 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 12 07:04:37 dist-test-slave-69rg krb5kdc[15268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518277, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM
I20260812 07:04:37.237222 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:37.222723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50194 (local address 127.6.61.62:35059)
0812 07:04:37.222962 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:37.222969 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:37.223203 (+ 234us) server_negotiation.cc:407] Connection header received
0812 07:04:37.223326 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:37.223329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:37.223391 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:37.223474 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:37.224679 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.225603 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.226890 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.227369 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.227468 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:37.231964 (+ 4496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:37.231991 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:37.231995 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:37.232032 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:37.234910 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:37.235579 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:37.235582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:37.235585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:37.235705 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:37.236045 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:37.236048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:37.236050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:37.236338 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:37.236561 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:37.236814 (+ 253us) server_negotiation.cc:299] Negotiation successful
0812 07:04:37.236990 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260812 07:04:37.241715 15326 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:04:37.248869 15326 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:04:37.275796 15904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:37.262059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:35663 (local address 127.0.0.1:33444)
0812 07:04:37.263102 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:37.263156 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:37.263339 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:37.265599 (+ 2260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:37.265608 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:37.265643 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:37.266191 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:37.266209 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.268678 (+ 2469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.268688 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:37.272444 (+ 3756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:37.272468 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.274627 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.274635 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:37.274907 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:37.274983 (+ 76us) client_negotiation.cc:769] Sending connection context
0812 07:04:37.275122 (+ 139us) client_negotiation.cc:240] Negotiation successful
0812 07:04:37.275345 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":4,"thread_start_us":943,"threads_started":1}
I20260812 07:04:37.276552 15905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:37.262786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33444 (local address 127.6.61.2:35663)
0812 07:04:37.264646 (+ 1860us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:37.264656 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:37.264686 (+ 30us) server_negotiation.cc:407] Connection header received
0812 07:04:37.264817 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:37.264828 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:37.265093 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:37.265322 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:37.266433 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.268463 (+ 2030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.272670 (+ 4207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.274076 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.274224 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:37.274473 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:37.276075 (+ 1602us) server_negotiation.cc:299] Negotiation successful
0812 07:04:37.276268 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1686,"thread_start_us":1558,"threads_started":1}
I20260812 07:04:37.276564 15906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:37.263227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60892 (local address 127.6.61.1:34611)
0812 07:04:37.263758 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:37.263767 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:37.263799 (+ 32us) server_negotiation.cc:407] Connection header received
0812 07:04:37.263943 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:37.263953 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:37.264278 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:37.265777 (+ 1499us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:37.266088 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.268217 (+ 2129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.273008 (+ 4791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.274601 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.274732 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:37.276042 (+ 1310us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:37.276109 (+ 67us) server_negotiation.cc:299] Negotiation successful
0812 07:04:37.276276 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":390,"threads_started":1}
I20260812 07:04:37.277098 15901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:37.262265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:34611 (local address 127.0.0.1:60892)
0812 07:04:37.263606 (+ 1341us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:37.263625 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:37.263709 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:37.264555 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:37.264565 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:37.264600 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:37.265114 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:37.265132 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.268430 (+ 3298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.268441 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:37.272805 (+ 4364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:37.272830 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.275304 (+ 2474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.275311 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:37.275488 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:37.275521 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 07:04:37.275598 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 07:04:37.275737 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":787,"thread_start_us":1157,"threads_started":1}
I20260812 07:04:37.281504 15643 tablet_service.cc:1467] Processing CreateTablet for tablet 380fda4b73a94bc392dbcee18e32971c (DEFAULT_TABLE table=test-table [id=47a9a5d5fb5a4db1923b674e7ba15642]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:04:37.281499 15481 tablet_service.cc:1467] Processing CreateTablet for tablet 380fda4b73a94bc392dbcee18e32971c (DEFAULT_TABLE table=test-table [id=47a9a5d5fb5a4db1923b674e7ba15642]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:04:37.283007 15481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 380fda4b73a94bc392dbcee18e32971c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:37.283028 15643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 380fda4b73a94bc392dbcee18e32971c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:37.284072 15907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:37.264460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:33203 (local address 127.0.0.1:51874)
0812 07:04:37.272590 (+ 8130us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:37.272616 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:37.272771 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:37.273932 (+ 1161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:37.273938 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:37.273968 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:37.274433 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:37.274449 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.277785 (+ 3336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.277793 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:37.279935 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:37.279950 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.283304 (+ 3354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.283311 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:37.283477 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:37.283513 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 07:04:37.283600 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 07:04:37.283735 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7998,"thread_start_us":7647,"threads_started":1}
I20260812 07:04:37.284994 15908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:37.265046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51874 (local address 127.6.61.3:33203)
0812 07:04:37.272104 (+ 7058us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:37.272115 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:37.272923 (+ 808us) server_negotiation.cc:407] Connection header received
0812 07:04:37.273042 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:37.273052 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:37.273358 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:37.273575 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:37.274713 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.277622 (+ 2909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.280135 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.282769 (+ 2634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.282907 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:37.283137 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:37.284469 (+ 1332us) server_negotiation.cc:299] Negotiation successful
0812 07:04:37.284643 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6840,"thread_start_us":6637,"threads_started":1}
I20260812 07:04:37.288887 15796 tablet_service.cc:1467] Processing CreateTablet for tablet 380fda4b73a94bc392dbcee18e32971c (DEFAULT_TABLE table=test-table [id=47a9a5d5fb5a4db1923b674e7ba15642]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:04:37.290565 15796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 380fda4b73a94bc392dbcee18e32971c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:37.303709 15910 tablet_bootstrap.cc:492] T 380fda4b73a94bc392dbcee18e32971c P 2da9547064764ae9bfb18d37ce320d7c: Bootstrap starting.
I20260812 07:04:37.307130 15909 tablet_bootstrap.cc:492] T 380fda4b73a94bc392dbcee18e32971c P e32364b73e254343b4bb6991207c2f85: Bootstrap starting.
I20260812 07:04:37.309343 15911 tablet_bootstrap.cc:492] T 380fda4b73a94bc392dbcee18e32971c P 59b3abeeb4124526b938e9dc4cf9a7c0: Bootstrap starting.
I20260812 07:04:37.310114 15910 tablet_bootstrap.cc:654] T 380fda4b73a94bc392dbcee18e32971c P 2da9547064764ae9bfb18d37ce320d7c: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:37.311610 15909 tablet_bootstrap.cc:654] T 380fda4b73a94bc392dbcee18e32971c P e32364b73e254343b4bb6991207c2f85: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:37.311669 15910 log.cc:824] T 380fda4b73a94bc392dbcee18e32971c P 2da9547064764ae9bfb18d37ce320d7c: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:37.312919 15911 tablet_bootstrap.cc:654] T 380fda4b73a94bc392dbcee18e32971c P 59b3abeeb4124526b938e9dc4cf9a7c0: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:37.312913 15909 log.cc:824] T 380fda4b73a94bc392dbcee18e32971c P e32364b73e254343b4bb6991207c2f85: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:37.314306 15911 log.cc:824] T 380fda4b73a94bc392dbcee18e32971c P 59b3abeeb4124526b938e9dc4cf9a7c0: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:37.315462 15910 tablet_bootstrap.cc:492] T 380fda4b73a94bc392dbcee18e32971c P 2da9547064764ae9bfb18d37ce320d7c: No bootstrap required, opened a new log
I20260812 07:04:37.315788 15910 ts_tablet_manager.cc:1397] T 380fda4b73a94bc392dbcee18e32971c P 2da9547064764ae9bfb18d37ce320d7c: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.005s
I20260812 07:04:37.316114 15909 tablet_bootstrap.cc:492] T 380fda4b73a94bc392dbcee18e32971c P e32364b73e254343b4bb6991207c2f85: No bootstrap required, opened a new log
I20260812 07:04:37.316416 15909 ts_tablet_manager.cc:1397] T 380fda4b73a94bc392dbcee18e32971c P e32364b73e254343b4bb6991207c2f85: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260812 07:04:37.317762 15911 tablet_bootstrap.cc:492] T 380fda4b73a94bc392dbcee18e32971c P 59b3abeeb4124526b938e9dc4cf9a7c0: No bootstrap required, opened a new log
I20260812 07:04:37.318145 15911 ts_tablet_manager.cc:1397] T 380fda4b73a94bc392dbcee18e32971c P 59b3abeeb4124526b938e9dc4cf9a7c0: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260812 07:04:37.327162 15911 raft_consensus.cc:348] T 380fda4b73a94bc392dbcee18e32971c P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:37.327591 15911 raft_consensus.cc:374] T 380fda4b73a94bc392dbcee18e32971c P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:37.327291 15909 raft_consensus.cc:348] T 380fda4b73a94bc392dbcee18e32971c P e32364b73e254343b4bb6991207c2f85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:37.327289 15910 raft_consensus.cc:348] T 380fda4b73a94bc392dbcee18e32971c P 2da9547064764ae9bfb18d37ce320d7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:37.327734 15910 raft_consensus.cc:374] T 380fda4b73a94bc392dbcee18e32971c P 2da9547064764ae9bfb18d37ce320d7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:37.327734 15909 raft_consensus.cc:374] T 380fda4b73a94bc392dbcee18e32971c P e32364b73e254343b4bb6991207c2f85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:37.327729 15911 raft_consensus.cc:729] T 380fda4b73a94bc392dbcee18e32971c P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59b3abeeb4124526b938e9dc4cf9a7c0, State: Initialized, Role: FOLLOWER
I20260812 07:04:37.327848 15909 raft_consensus.cc:729] T 380fda4b73a94bc392dbcee18e32971c P e32364b73e254343b4bb6991207c2f85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e32364b73e254343b4bb6991207c2f85, State: Initialized, Role: FOLLOWER
I20260812 07:04:37.327857 15910 raft_consensus.cc:729] T 380fda4b73a94bc392dbcee18e32971c P 2da9547064764ae9bfb18d37ce320d7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2da9547064764ae9bfb18d37ce320d7c, State: Initialized, Role: FOLLOWER
I20260812 07:04:37.328375 15911 consensus_queue.cc:260] T 380fda4b73a94bc392dbcee18e32971c P 59b3abeeb4124526b938e9dc4cf9a7c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:37.328549 15909 consensus_queue.cc:260] T 380fda4b73a94bc392dbcee18e32971c P e32364b73e254343b4bb6991207c2f85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:37.328555 15910 consensus_queue.cc:260] T 380fda4b73a94bc392dbcee18e32971c P 2da9547064764ae9bfb18d37ce320d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:37.329882 15716 heartbeater.cc:503] Master 127.6.61.62:35059 was elected leader, sending a full tablet report...
I20260812 07:04:37.332175 15909 ts_tablet_manager.cc:1428] T 380fda4b73a94bc392dbcee18e32971c P e32364b73e254343b4bb6991207c2f85: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
I20260812 07:04:37.332207 15910 ts_tablet_manager.cc:1428] T 380fda4b73a94bc392dbcee18e32971c P 2da9547064764ae9bfb18d37ce320d7c: Time spent starting tablet: real 0.016s user 0.008s sys 0.006s
I20260812 07:04:37.333151 15882 heartbeater.cc:503] Master 127.6.61.62:35059 was elected leader, sending a full tablet report...
I20260812 07:04:37.334854 15911 ts_tablet_manager.cc:1428] T 380fda4b73a94bc392dbcee18e32971c P 59b3abeeb4124526b938e9dc4cf9a7c0: Time spent starting tablet: real 0.016s user 0.008s sys 0.005s
W20260812 07:04:37.335335 15717 tablet.cc:2369] T 380fda4b73a94bc392dbcee18e32971c P e32364b73e254343b4bb6991207c2f85: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 07:04:37.404294 15884 tablet.cc:2369] T 380fda4b73a94bc392dbcee18e32971c P 59b3abeeb4124526b938e9dc4cf9a7c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 07:04:37.492760 15562 tablet.cc:2369] T 380fda4b73a94bc392dbcee18e32971c P 2da9547064764ae9bfb18d37ce320d7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:04:37.578994 15917 raft_consensus.cc:482] T 380fda4b73a94bc392dbcee18e32971c P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:04:37.579386 15917 raft_consensus.cc:504] T 380fda4b73a94bc392dbcee18e32971c P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:37.581115 15917 leader_election.cc:288] T 380fda4b73a94bc392dbcee18e32971c P 59b3abeeb4124526b938e9dc4cf9a7c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e32364b73e254343b4bb6991207c2f85 (127.6.61.2:35663), 2da9547064764ae9bfb18d37ce320d7c (127.6.61.1:34611)
I20260812 07:04:37.589082 15905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:37.582470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:34461 (local address 127.6.61.2:35663)
0812 07:04:37.582736 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:37.582743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:37.582762 (+ 19us) server_negotiation.cc:407] Connection header received
0812 07:04:37.582838 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:37.582844 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:37.582923 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:37.583030 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:37.583509 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.584697 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.587291 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.588113 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.588275 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:37.588392 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:37.588659 (+ 267us) server_negotiation.cc:299] Negotiation successful
0812 07:04:37.588799 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260812 07:04:37.589078 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:37.581422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:35663 (local address 127.6.61.3:34461)
0812 07:04:37.581939 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:37.581961 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:37.582089 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:37.583051 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:37.583055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:37.583083 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:37.583317 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:37.583333 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.584854 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.584862 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:37.587063 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:37.587079 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.588290 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.588296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:37.588489 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:37.588514 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 07:04:37.588620 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 07:04:37.588759 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":51}
I20260812 07:04:37.589738 15906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:37.581764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:46623 (local address 127.6.61.1:34611)
0812 07:04:37.582050 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:37.582059 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:37.582738 (+ 679us) server_negotiation.cc:407] Connection header received
0812 07:04:37.582904 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:37.582910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:37.582998 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:37.583083 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:37.583775 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.584499 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.587243 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.588513 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.588657 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:37.588758 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:37.589308 (+ 550us) server_negotiation.cc:299] Negotiation successful
0812 07:04:37.589472 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260812 07:04:37.590484 15920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:37.581531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:34611 (local address 127.6.61.3:46623)
0812 07:04:37.582609 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:37.582627 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:37.582759 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:37.583208 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:37.583218 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:37.583242 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:37.583611 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:37.583619 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.584651 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.584656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:37.587062 (+ 2406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:37.587079 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.588918 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.588924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:37.589105 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:37.589139 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 07:04:37.589936 (+ 797us) client_negotiation.cc:240] Negotiation successful
0812 07:04:37.590096 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":965,"mutex_wait_us":149,"thread_start_us":802,"threads_started":1}
I20260812 07:04:37.590478 15665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380fda4b73a94bc392dbcee18e32971c" candidate_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e32364b73e254343b4bb6991207c2f85" is_pre_election: true
I20260812 07:04:37.591077 15665 raft_consensus.cc:2454] T 380fda4b73a94bc392dbcee18e32971c P e32364b73e254343b4bb6991207c2f85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59b3abeeb4124526b938e9dc4cf9a7c0 in term 0.
I20260812 07:04:37.591709 15503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380fda4b73a94bc392dbcee18e32971c" candidate_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2da9547064764ae9bfb18d37ce320d7c" is_pre_election: true
I20260812 07:04:37.591871 15742 leader_election.cc:302] T 380fda4b73a94bc392dbcee18e32971c P 59b3abeeb4124526b938e9dc4cf9a7c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59b3abeeb4124526b938e9dc4cf9a7c0, e32364b73e254343b4bb6991207c2f85; no voters:
I20260812 07:04:37.592288 15917 raft_consensus.cc:2781] T 380fda4b73a94bc392dbcee18e32971c P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:04:37.592244 15503 raft_consensus.cc:2454] T 380fda4b73a94bc392dbcee18e32971c P 2da9547064764ae9bfb18d37ce320d7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59b3abeeb4124526b938e9dc4cf9a7c0 in term 0.
I20260812 07:04:37.592453 15917 raft_consensus.cc:482] T 380fda4b73a94bc392dbcee18e32971c P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:04:37.592573 15917 raft_consensus.cc:3037] T 380fda4b73a94bc392dbcee18e32971c P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:37.595072 15917 raft_consensus.cc:504] T 380fda4b73a94bc392dbcee18e32971c P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:37.595702 15917 leader_election.cc:288] T 380fda4b73a94bc392dbcee18e32971c P 59b3abeeb4124526b938e9dc4cf9a7c0 [CANDIDATE]: Term 1 election: Requested vote from peers e32364b73e254343b4bb6991207c2f85 (127.6.61.2:35663), 2da9547064764ae9bfb18d37ce320d7c (127.6.61.1:34611)
I20260812 07:04:37.596192 15665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380fda4b73a94bc392dbcee18e32971c" candidate_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e32364b73e254343b4bb6991207c2f85"
I20260812 07:04:37.596248 15503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380fda4b73a94bc392dbcee18e32971c" candidate_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2da9547064764ae9bfb18d37ce320d7c"
I20260812 07:04:37.596412 15665 raft_consensus.cc:3037] T 380fda4b73a94bc392dbcee18e32971c P e32364b73e254343b4bb6991207c2f85 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:37.596457 15503 raft_consensus.cc:3037] T 380fda4b73a94bc392dbcee18e32971c P 2da9547064764ae9bfb18d37ce320d7c [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:37.598785 15665 raft_consensus.cc:2454] T 380fda4b73a94bc392dbcee18e32971c P e32364b73e254343b4bb6991207c2f85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59b3abeeb4124526b938e9dc4cf9a7c0 in term 1.
I20260812 07:04:37.599037 15503 raft_consensus.cc:2454] T 380fda4b73a94bc392dbcee18e32971c P 2da9547064764ae9bfb18d37ce320d7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59b3abeeb4124526b938e9dc4cf9a7c0 in term 1.
I20260812 07:04:37.599222 15742 leader_election.cc:302] T 380fda4b73a94bc392dbcee18e32971c P 59b3abeeb4124526b938e9dc4cf9a7c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59b3abeeb4124526b938e9dc4cf9a7c0, e32364b73e254343b4bb6991207c2f85; no voters:
I20260812 07:04:37.599527 15917 raft_consensus.cc:2781] T 380fda4b73a94bc392dbcee18e32971c P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:04:37.599951 15917 raft_consensus.cc:686] T 380fda4b73a94bc392dbcee18e32971c P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 1 LEADER]: Becoming Leader. State: Replica: 59b3abeeb4124526b938e9dc4cf9a7c0, State: Running, Role: LEADER
I20260812 07:04:37.600419 15917 consensus_queue.cc:237] T 380fda4b73a94bc392dbcee18e32971c P 59b3abeeb4124526b938e9dc4cf9a7c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:37.605445 15324 catalog_manager.cc:5495] T 380fda4b73a94bc392dbcee18e32971c P 59b3abeeb4124526b938e9dc4cf9a7c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59b3abeeb4124526b938e9dc4cf9a7c0 (127.6.61.3). New cstate: current_term: 1 leader_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } health_report { overall_health: HEALTHY } } }
I20260812 07:04:37.648910 15908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:37.643644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51876 (local address 127.6.61.3:33203)
0812 07:04:37.643830 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:37.643835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:37.643873 (+ 38us) server_negotiation.cc:407] Connection header received
0812 07:04:37.644156 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:37.644161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:37.644256 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:37.644406 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:04:37.644764 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.645451 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.646812 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.647231 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.647327 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:37.647791 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:04:37.648368 (+ 577us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:04:37.648489 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:37.648632 (+ 143us) server_negotiation.cc:299] Negotiation successful
0812 07:04:37.648704 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 07:04:37.657220 15665 raft_consensus.cc:1261] T 380fda4b73a94bc392dbcee18e32971c P e32364b73e254343b4bb6991207c2f85 [term 1 FOLLOWER]: Refusing update from remote peer 59b3abeeb4124526b938e9dc4cf9a7c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:04:37.657251 15503 raft_consensus.cc:1261] T 380fda4b73a94bc392dbcee18e32971c P 2da9547064764ae9bfb18d37ce320d7c [term 1 FOLLOWER]: Refusing update from remote peer 59b3abeeb4124526b938e9dc4cf9a7c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:04:37.658334 15921 consensus_queue.cc:1035] T 380fda4b73a94bc392dbcee18e32971c P 59b3abeeb4124526b938e9dc4cf9a7c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:04:37.658739 15917 consensus_queue.cc:1035] T 380fda4b73a94bc392dbcee18e32971c P 59b3abeeb4124526b938e9dc4cf9a7c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:04:37.685894 15326 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:04:37.686179 15326 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:04:37.689859 15326 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc0633cfe3c74aa6a8a6d4fbe3cec982: Sending DeleteTablet for 3 replicas of tablet 380fda4b73a94bc392dbcee18e32971c
I20260812 07:04:37.691058 15481 tablet_service.cc:1514] Processing DeleteTablet for tablet 380fda4b73a94bc392dbcee18e32971c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:04:37 UTC) from {username='oryx', principal='oryx/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:60892
I20260812 07:04:37.691124 15643 tablet_service.cc:1514] Processing DeleteTablet for tablet 380fda4b73a94bc392dbcee18e32971c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:04:37 UTC) from {username='oryx', principal='oryx/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:33444
I20260812 07:04:37.691393 15796 tablet_service.cc:1514] Processing DeleteTablet for tablet 380fda4b73a94bc392dbcee18e32971c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:04:37 UTC) from {username='oryx', principal='oryx/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:51874
I20260812 07:04:37.693567 15326 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:04:37.694650 15326 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:04:37.705854 15939 tablet_replica.cc:330] T 380fda4b73a94bc392dbcee18e32971c P 2da9547064764ae9bfb18d37ce320d7c: stopping tablet replica
I20260812 07:04:37.707070 15481 tablet_service.cc:1467] Processing CreateTablet for tablet 176f91a5f60b45db872fbdfbb0c3fcb3 (DEFAULT_TABLE table=test-table [id=dcd77504f7f342578e62b876ff9fc1c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:04:37.707430 15641 tablet_service.cc:1467] Processing CreateTablet for tablet 176f91a5f60b45db872fbdfbb0c3fcb3 (DEFAULT_TABLE table=test-table [id=dcd77504f7f342578e62b876ff9fc1c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:04:37.707839 15481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 176f91a5f60b45db872fbdfbb0c3fcb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:37.708112 15641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 176f91a5f60b45db872fbdfbb0c3fcb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:37.710944 15795 tablet_service.cc:1467] Processing CreateTablet for tablet 176f91a5f60b45db872fbdfbb0c3fcb3 (DEFAULT_TABLE table=test-table [id=dcd77504f7f342578e62b876ff9fc1c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:04:37.711725 15795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 176f91a5f60b45db872fbdfbb0c3fcb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:37.714085 15939 raft_consensus.cc:2229] T 380fda4b73a94bc392dbcee18e32971c P 2da9547064764ae9bfb18d37ce320d7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:04:37.714735 15939 raft_consensus.cc:2258] T 380fda4b73a94bc392dbcee18e32971c P 2da9547064764ae9bfb18d37ce320d7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:04:37.716701 15938 tablet_replica.cc:330] T 380fda4b73a94bc392dbcee18e32971c P e32364b73e254343b4bb6991207c2f85: stopping tablet replica
I20260812 07:04:37.717059 15910 tablet_bootstrap.cc:492] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 2da9547064764ae9bfb18d37ce320d7c: Bootstrap starting.
I20260812 07:04:37.718940 15910 tablet_bootstrap.cc:654] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 2da9547064764ae9bfb18d37ce320d7c: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:37.721326 15911 tablet_bootstrap.cc:492] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 59b3abeeb4124526b938e9dc4cf9a7c0: Bootstrap starting.
I20260812 07:04:37.721980 15939 ts_tablet_manager.cc:1905] T 380fda4b73a94bc392dbcee18e32971c P 2da9547064764ae9bfb18d37ce320d7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:04:37.723513 15911 tablet_bootstrap.cc:654] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 59b3abeeb4124526b938e9dc4cf9a7c0: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:37.717347 15909 tablet_bootstrap.cc:492] T 176f91a5f60b45db872fbdfbb0c3fcb3 P e32364b73e254343b4bb6991207c2f85: Bootstrap starting.
I20260812 07:04:37.728988 15938 raft_consensus.cc:2229] T 380fda4b73a94bc392dbcee18e32971c P e32364b73e254343b4bb6991207c2f85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:04:37.729331 15940 tablet_replica.cc:330] T 380fda4b73a94bc392dbcee18e32971c P 59b3abeeb4124526b938e9dc4cf9a7c0: stopping tablet replica
I20260812 07:04:37.729604 15938 raft_consensus.cc:2258] T 380fda4b73a94bc392dbcee18e32971c P e32364b73e254343b4bb6991207c2f85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:04:37.729746 15940 raft_consensus.cc:2229] T 380fda4b73a94bc392dbcee18e32971c P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:04:37.730885 15909 tablet_bootstrap.cc:654] T 176f91a5f60b45db872fbdfbb0c3fcb3 P e32364b73e254343b4bb6991207c2f85: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:37.732846 15938 ts_tablet_manager.cc:1905] T 380fda4b73a94bc392dbcee18e32971c P e32364b73e254343b4bb6991207c2f85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:04:37.739312 15909 tablet_bootstrap.cc:492] T 176f91a5f60b45db872fbdfbb0c3fcb3 P e32364b73e254343b4bb6991207c2f85: No bootstrap required, opened a new log
I20260812 07:04:37.739497 15909 ts_tablet_manager.cc:1397] T 176f91a5f60b45db872fbdfbb0c3fcb3 P e32364b73e254343b4bb6991207c2f85: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20260812 07:04:37.740420 15938 ts_tablet_manager.cc:1918] T 380fda4b73a94bc392dbcee18e32971c P e32364b73e254343b4bb6991207c2f85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:04:37.740378 15909 raft_consensus.cc:348] T 176f91a5f60b45db872fbdfbb0c3fcb3 P e32364b73e254343b4bb6991207c2f85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:37.740573 15938 log.cc:1196] T 380fda4b73a94bc392dbcee18e32971c P e32364b73e254343b4bb6991207c2f85: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-1/wal/wals/380fda4b73a94bc392dbcee18e32971c
I20260812 07:04:37.740633 15909 raft_consensus.cc:374] T 176f91a5f60b45db872fbdfbb0c3fcb3 P e32364b73e254343b4bb6991207c2f85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:37.740752 15909 raft_consensus.cc:729] T 176f91a5f60b45db872fbdfbb0c3fcb3 P e32364b73e254343b4bb6991207c2f85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e32364b73e254343b4bb6991207c2f85, State: Initialized, Role: FOLLOWER
I20260812 07:04:37.741197 15938 ts_tablet_manager.cc:1939] T 380fda4b73a94bc392dbcee18e32971c P e32364b73e254343b4bb6991207c2f85: Deleting consensus metadata
I20260812 07:04:37.741071 15909 consensus_queue.cc:260] T 176f91a5f60b45db872fbdfbb0c3fcb3 P e32364b73e254343b4bb6991207c2f85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:37.743217 15909 ts_tablet_manager.cc:1428] T 176f91a5f60b45db872fbdfbb0c3fcb3 P e32364b73e254343b4bb6991207c2f85: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260812 07:04:37.743103 15309 catalog_manager.cc:4841] TS e32364b73e254343b4bb6991207c2f85 (127.6.61.2:35663): tablet 380fda4b73a94bc392dbcee18e32971c (table test-table [id=47a9a5d5fb5a4db1923b674e7ba15642]) successfully deleted
I20260812 07:04:37.744167 15940 raft_consensus.cc:2258] T 380fda4b73a94bc392dbcee18e32971c P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:04:37.744691 15910 tablet_bootstrap.cc:492] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 2da9547064764ae9bfb18d37ce320d7c: No bootstrap required, opened a new log
I20260812 07:04:37.744932 15910 ts_tablet_manager.cc:1397] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 2da9547064764ae9bfb18d37ce320d7c: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.005s
I20260812 07:04:37.745893 15910 raft_consensus.cc:348] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 2da9547064764ae9bfb18d37ce320d7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:37.746201 15910 raft_consensus.cc:374] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 2da9547064764ae9bfb18d37ce320d7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:37.746371 15910 raft_consensus.cc:729] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 2da9547064764ae9bfb18d37ce320d7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2da9547064764ae9bfb18d37ce320d7c, State: Initialized, Role: FOLLOWER
I20260812 07:04:37.746749 15940 ts_tablet_manager.cc:1905] T 380fda4b73a94bc392dbcee18e32971c P 59b3abeeb4124526b938e9dc4cf9a7c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:04:37.746737 15910 consensus_queue.cc:260] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 2da9547064764ae9bfb18d37ce320d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:37.748098 15910 ts_tablet_manager.cc:1428] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 2da9547064764ae9bfb18d37ce320d7c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 07:04:37.748805 15939 ts_tablet_manager.cc:1918] T 380fda4b73a94bc392dbcee18e32971c P 2da9547064764ae9bfb18d37ce320d7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:04:37.749028 15939 log.cc:1196] T 380fda4b73a94bc392dbcee18e32971c P 2da9547064764ae9bfb18d37ce320d7c: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-0/wal/wals/380fda4b73a94bc392dbcee18e32971c
I20260812 07:04:37.749667 15939 ts_tablet_manager.cc:1939] T 380fda4b73a94bc392dbcee18e32971c P 2da9547064764ae9bfb18d37ce320d7c: Deleting consensus metadata
I20260812 07:04:37.751585 15305 catalog_manager.cc:4841] TS 2da9547064764ae9bfb18d37ce320d7c (127.6.61.1:34611): tablet 380fda4b73a94bc392dbcee18e32971c (table test-table [id=47a9a5d5fb5a4db1923b674e7ba15642]) successfully deleted
I20260812 07:04:37.753495 15911 tablet_bootstrap.cc:492] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 59b3abeeb4124526b938e9dc4cf9a7c0: No bootstrap required, opened a new log
I20260812 07:04:37.753685 15911 ts_tablet_manager.cc:1397] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 59b3abeeb4124526b938e9dc4cf9a7c0: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.003s
I20260812 07:04:37.754648 15911 raft_consensus.cc:348] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:37.754913 15911 raft_consensus.cc:374] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:37.755030 15940 ts_tablet_manager.cc:1918] T 380fda4b73a94bc392dbcee18e32971c P 59b3abeeb4124526b938e9dc4cf9a7c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:04:37.755039 15911 raft_consensus.cc:729] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59b3abeeb4124526b938e9dc4cf9a7c0, State: Initialized, Role: FOLLOWER
I20260812 07:04:37.755234 15940 log.cc:1196] T 380fda4b73a94bc392dbcee18e32971c P 59b3abeeb4124526b938e9dc4cf9a7c0: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-2/wal/wals/380fda4b73a94bc392dbcee18e32971c
I20260812 07:04:37.755426 15911 consensus_queue.cc:260] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 59b3abeeb4124526b938e9dc4cf9a7c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:37.755951 15940 ts_tablet_manager.cc:1939] T 380fda4b73a94bc392dbcee18e32971c P 59b3abeeb4124526b938e9dc4cf9a7c0: Deleting consensus metadata
I20260812 07:04:37.756279 15911 ts_tablet_manager.cc:1428] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 59b3abeeb4124526b938e9dc4cf9a7c0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 07:04:37.757718 15307 catalog_manager.cc:4841] TS 59b3abeeb4124526b938e9dc4cf9a7c0 (127.6.61.3:33203): tablet 380fda4b73a94bc392dbcee18e32971c (table test-table [id=47a9a5d5fb5a4db1923b674e7ba15642]) successfully deleted
I20260812 07:04:37.894508 15919 raft_consensus.cc:482] T 176f91a5f60b45db872fbdfbb0c3fcb3 P e32364b73e254343b4bb6991207c2f85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:04:37.894794 15919 raft_consensus.cc:504] T 176f91a5f60b45db872fbdfbb0c3fcb3 P e32364b73e254343b4bb6991207c2f85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:37.896806 15919 leader_election.cc:288] T 176f91a5f60b45db872fbdfbb0c3fcb3 P e32364b73e254343b4bb6991207c2f85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2da9547064764ae9bfb18d37ce320d7c (127.6.61.1:34611), 59b3abeeb4124526b938e9dc4cf9a7c0 (127.6.61.3:33203)
I20260812 07:04:37.904975 15908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:37.897196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:54153 (local address 127.6.61.3:33203)
0812 07:04:37.897450 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:37.897459 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:37.898456 (+ 997us) server_negotiation.cc:407] Connection header received
0812 07:04:37.898707 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:37.898713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:37.898793 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:37.898897 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:37.899598 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.900769 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.902768 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.903848 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.903986 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:37.904099 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:37.904476 (+ 377us) server_negotiation.cc:299] Negotiation successful
0812 07:04:37.904633 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260812 07:04:37.905014 15945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:37.897360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:33203 (local address 127.6.61.2:54153)
0812 07:04:37.898361 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:37.898379 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:37.898551 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:37.898920 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:37.898925 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:37.898957 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:37.899409 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:37.899422 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.900905 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.900916 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:37.902577 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:37.902590 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.904073 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.904082 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:37.904271 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:37.904312 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 07:04:37.904415 (+ 103us) client_negotiation.cc:240] Negotiation successful
0812 07:04:37.904562 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"mutex_wait_us":24,"thread_start_us":649,"threads_started":1}
I20260812 07:04:37.905792 15906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:37.897747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:56709 (local address 127.6.61.1:34611)
0812 07:04:37.897959 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:37.897965 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:37.898136 (+ 171us) server_negotiation.cc:407] Connection header received
0812 07:04:37.898490 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:37.898497 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:37.898606 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:37.898718 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:37.899428 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.900606 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.903334 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.904542 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.904662 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:37.904765 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:37.905369 (+ 604us) server_negotiation.cc:299] Negotiation successful
0812 07:04:37.905501 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260812 07:04:37.905855 15944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:37.897145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:34611 (local address 127.6.61.2:56709)
0812 07:04:37.897992 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:37.898013 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:37.898297 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:37.898743 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:37.898748 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:37.898772 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:37.899226 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:37.899240 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.900756 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.900762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:37.903152 (+ 2390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:37.903168 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:37.904939 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:37.904947 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:37.905164 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:37.905200 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 07:04:37.905317 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 07:04:37.905450 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":73,"thread_start_us":510,"threads_started":1}
I20260812 07:04:37.906685 15503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "176f91a5f60b45db872fbdfbb0c3fcb3" candidate_uuid: "e32364b73e254343b4bb6991207c2f85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2da9547064764ae9bfb18d37ce320d7c" is_pre_election: true
I20260812 07:04:37.906632 15827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "176f91a5f60b45db872fbdfbb0c3fcb3" candidate_uuid: "e32364b73e254343b4bb6991207c2f85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" is_pre_election: true
I20260812 07:04:37.906970 15503 raft_consensus.cc:2454] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 2da9547064764ae9bfb18d37ce320d7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e32364b73e254343b4bb6991207c2f85 in term 0.
I20260812 07:04:37.907275 15827 raft_consensus.cc:2454] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e32364b73e254343b4bb6991207c2f85 in term 0.
I20260812 07:04:37.907492 15582 leader_election.cc:302] T 176f91a5f60b45db872fbdfbb0c3fcb3 P e32364b73e254343b4bb6991207c2f85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2da9547064764ae9bfb18d37ce320d7c, e32364b73e254343b4bb6991207c2f85; no voters:
I20260812 07:04:37.907974 15919 raft_consensus.cc:2781] T 176f91a5f60b45db872fbdfbb0c3fcb3 P e32364b73e254343b4bb6991207c2f85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:04:37.908113 15919 raft_consensus.cc:482] T 176f91a5f60b45db872fbdfbb0c3fcb3 P e32364b73e254343b4bb6991207c2f85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:04:37.908217 15919 raft_consensus.cc:3037] T 176f91a5f60b45db872fbdfbb0c3fcb3 P e32364b73e254343b4bb6991207c2f85 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:37.911568 15919 raft_consensus.cc:504] T 176f91a5f60b45db872fbdfbb0c3fcb3 P e32364b73e254343b4bb6991207c2f85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:37.912292 15919 leader_election.cc:288] T 176f91a5f60b45db872fbdfbb0c3fcb3 P e32364b73e254343b4bb6991207c2f85 [CANDIDATE]: Term 1 election: Requested vote from peers 2da9547064764ae9bfb18d37ce320d7c (127.6.61.1:34611), 59b3abeeb4124526b938e9dc4cf9a7c0 (127.6.61.3:33203)
I20260812 07:04:37.912817 15827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "176f91a5f60b45db872fbdfbb0c3fcb3" candidate_uuid: "e32364b73e254343b4bb6991207c2f85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0"
I20260812 07:04:37.912812 15503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "176f91a5f60b45db872fbdfbb0c3fcb3" candidate_uuid: "e32364b73e254343b4bb6991207c2f85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2da9547064764ae9bfb18d37ce320d7c"
I20260812 07:04:37.913009 15503 raft_consensus.cc:3037] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 2da9547064764ae9bfb18d37ce320d7c [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:37.913033 15827 raft_consensus.cc:3037] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:37.915518 15827 raft_consensus.cc:2454] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e32364b73e254343b4bb6991207c2f85 in term 1.
I20260812 07:04:37.915881 15503 raft_consensus.cc:2454] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 2da9547064764ae9bfb18d37ce320d7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e32364b73e254343b4bb6991207c2f85 in term 1.
I20260812 07:04:37.916002 15584 leader_election.cc:302] T 176f91a5f60b45db872fbdfbb0c3fcb3 P e32364b73e254343b4bb6991207c2f85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59b3abeeb4124526b938e9dc4cf9a7c0, e32364b73e254343b4bb6991207c2f85; no voters:
I20260812 07:04:37.916316 15919 raft_consensus.cc:2781] T 176f91a5f60b45db872fbdfbb0c3fcb3 P e32364b73e254343b4bb6991207c2f85 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:04:37.916821 15919 raft_consensus.cc:686] T 176f91a5f60b45db872fbdfbb0c3fcb3 P e32364b73e254343b4bb6991207c2f85 [term 1 LEADER]: Becoming Leader. State: Replica: e32364b73e254343b4bb6991207c2f85, State: Running, Role: LEADER
I20260812 07:04:37.917240 15919 consensus_queue.cc:237] T 176f91a5f60b45db872fbdfbb0c3fcb3 P e32364b73e254343b4bb6991207c2f85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:37.921109 15324 catalog_manager.cc:5495] T 176f91a5f60b45db872fbdfbb0c3fcb3 P e32364b73e254343b4bb6991207c2f85 reported cstate change: term changed from 0 to 1, leader changed from <none> to e32364b73e254343b4bb6991207c2f85 (127.6.61.2). New cstate: current_term: 1 leader_uuid: "e32364b73e254343b4bb6991207c2f85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } health_report { overall_health: UNKNOWN } } }
Aug 12 07:04:38 dist-test-slave-69rg krb5kdc[15268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518275, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.62@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM
I20260812 07:04:38.023109 15956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:37.988799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50200 (local address 127.6.61.62:35059)
0812 07:04:38.005246 (+ 16447us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:38.005253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:38.005929 (+ 676us) server_negotiation.cc:407] Connection header received
0812 07:04:38.006087 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:38.006092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:38.006185 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:38.006321 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:38.007798 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:38.009067 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:38.011001 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:38.011700 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:38.011844 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:38.016536 (+ 4692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:38.016543 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:38.016548 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:38.016588 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:38.019516 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:38.020549 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:38.020554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:38.020556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:38.020716 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:38.021244 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:38.021250 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:38.021253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:38.021716 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:38.021895 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:38.022418 (+ 523us) server_negotiation.cc:299] Negotiation successful
0812 07:04:38.022815 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":156,"server-negotiator.queue_time_us":16250,"thread_start_us":16954,"threads_started":1}
I20260812 07:04:38.023157 15957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:37.988764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:35059 (local address 127.0.0.1:50200)
0812 07:04:38.005689 (+ 16925us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:38.005712 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:38.005925 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:38.006383 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:38.006389 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:38.007371 (+ 982us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:38.007628 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:38.007640 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:38.009223 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:38.009230 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:38.010839 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:38.010855 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:38.011900 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:38.011908 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:38.012106 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:38.012988 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:38.013016 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:38.016281 (+ 3265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:38.019675 (+ 3394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:38.019724 (+ 49us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:38.019734 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:38.020416 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:38.020897 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:38.020902 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:38.020907 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:38.021106 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:38.021886 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:38.021897 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:38.022110 (+ 213us) client_negotiation.cc:769] Sending connection context
0812 07:04:38.022311 (+ 201us) client_negotiation.cc:240] Negotiation successful
0812 07:04:38.022658 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16764,"thread_start_us":16639,"threads_started":1}
I20260812 07:04:38.029057 15324 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:50200:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 07:04:38.030215 15324 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:04:38.037902 15641 tablet_service.cc:1467] Processing CreateTablet for tablet 35153c14a1e144d28ca5fa4e587a1ec5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6b674fc15f994c56be48c957d7b09fee]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 07:04:38.037854 15481 tablet_service.cc:1467] Processing CreateTablet for tablet 35153c14a1e144d28ca5fa4e587a1ec5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6b674fc15f994c56be48c957d7b09fee]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 07:04:38.038482 15641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35153c14a1e144d28ca5fa4e587a1ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:38.038502 15481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35153c14a1e144d28ca5fa4e587a1ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:38.045066 15795 tablet_service.cc:1467] Processing CreateTablet for tablet 35153c14a1e144d28ca5fa4e587a1ec5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6b674fc15f994c56be48c957d7b09fee]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 07:04:38.045665 15795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35153c14a1e144d28ca5fa4e587a1ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:38.045969 15910 tablet_bootstrap.cc:492] T 35153c14a1e144d28ca5fa4e587a1ec5 P 2da9547064764ae9bfb18d37ce320d7c: Bootstrap starting.
I20260812 07:04:38.046253 15909 tablet_bootstrap.cc:492] T 35153c14a1e144d28ca5fa4e587a1ec5 P e32364b73e254343b4bb6991207c2f85: Bootstrap starting.
I20260812 07:04:38.047747 15910 tablet_bootstrap.cc:654] T 35153c14a1e144d28ca5fa4e587a1ec5 P 2da9547064764ae9bfb18d37ce320d7c: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:38.047907 15909 tablet_bootstrap.cc:654] T 35153c14a1e144d28ca5fa4e587a1ec5 P e32364b73e254343b4bb6991207c2f85: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:38.055305 15911 tablet_bootstrap.cc:492] T 35153c14a1e144d28ca5fa4e587a1ec5 P 59b3abeeb4124526b938e9dc4cf9a7c0: Bootstrap starting.
I20260812 07:04:38.057480 15911 tablet_bootstrap.cc:654] T 35153c14a1e144d28ca5fa4e587a1ec5 P 59b3abeeb4124526b938e9dc4cf9a7c0: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:38.057726 15909 tablet_bootstrap.cc:492] T 35153c14a1e144d28ca5fa4e587a1ec5 P e32364b73e254343b4bb6991207c2f85: No bootstrap required, opened a new log
I20260812 07:04:38.057940 15909 ts_tablet_manager.cc:1397] T 35153c14a1e144d28ca5fa4e587a1ec5 P e32364b73e254343b4bb6991207c2f85: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.009s
I20260812 07:04:38.059073 15910 tablet_bootstrap.cc:492] T 35153c14a1e144d28ca5fa4e587a1ec5 P 2da9547064764ae9bfb18d37ce320d7c: No bootstrap required, opened a new log
I20260812 07:04:38.059319 15910 ts_tablet_manager.cc:1397] T 35153c14a1e144d28ca5fa4e587a1ec5 P 2da9547064764ae9bfb18d37ce320d7c: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20260812 07:04:38.060249 15910 raft_consensus.cc:348] T 35153c14a1e144d28ca5fa4e587a1ec5 P 2da9547064764ae9bfb18d37ce320d7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:38.060603 15910 raft_consensus.cc:374] T 35153c14a1e144d28ca5fa4e587a1ec5 P 2da9547064764ae9bfb18d37ce320d7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:38.060734 15910 raft_consensus.cc:729] T 35153c14a1e144d28ca5fa4e587a1ec5 P 2da9547064764ae9bfb18d37ce320d7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2da9547064764ae9bfb18d37ce320d7c, State: Initialized, Role: FOLLOWER
I20260812 07:04:38.061049 15910 consensus_queue.cc:260] T 35153c14a1e144d28ca5fa4e587a1ec5 P 2da9547064764ae9bfb18d37ce320d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:38.061661 15918 tablet_replica.cc:439] T 35153c14a1e144d28ca5fa4e587a1ec5 P 2da9547064764ae9bfb18d37ce320d7c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } } }
I20260812 07:04:38.062148 15918 tablet_replica.cc:442] T 35153c14a1e144d28ca5fa4e587a1ec5 P 2da9547064764ae9bfb18d37ce320d7c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:04:38.062738 15910 ts_tablet_manager.cc:1428] T 35153c14a1e144d28ca5fa4e587a1ec5 P 2da9547064764ae9bfb18d37ce320d7c: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260812 07:04:38.058840 15909 raft_consensus.cc:348] T 35153c14a1e144d28ca5fa4e587a1ec5 P e32364b73e254343b4bb6991207c2f85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:38.064237 15909 raft_consensus.cc:374] T 35153c14a1e144d28ca5fa4e587a1ec5 P e32364b73e254343b4bb6991207c2f85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:38.064466 15909 raft_consensus.cc:729] T 35153c14a1e144d28ca5fa4e587a1ec5 P e32364b73e254343b4bb6991207c2f85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e32364b73e254343b4bb6991207c2f85, State: Initialized, Role: FOLLOWER
I20260812 07:04:38.065016 15909 consensus_queue.cc:260] T 35153c14a1e144d28ca5fa4e587a1ec5 P e32364b73e254343b4bb6991207c2f85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:38.065912 15919 tablet_replica.cc:439] T 35153c14a1e144d28ca5fa4e587a1ec5 P e32364b73e254343b4bb6991207c2f85: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } } }
I20260812 07:04:38.066522 15919 tablet_replica.cc:442] T 35153c14a1e144d28ca5fa4e587a1ec5 P e32364b73e254343b4bb6991207c2f85: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:04:38.067512 15909 ts_tablet_manager.cc:1428] T 35153c14a1e144d28ca5fa4e587a1ec5 P e32364b73e254343b4bb6991207c2f85: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20260812 07:04:38.071805 15911 tablet_bootstrap.cc:492] T 35153c14a1e144d28ca5fa4e587a1ec5 P 59b3abeeb4124526b938e9dc4cf9a7c0: No bootstrap required, opened a new log
I20260812 07:04:38.072026 15911 ts_tablet_manager.cc:1397] T 35153c14a1e144d28ca5fa4e587a1ec5 P 59b3abeeb4124526b938e9dc4cf9a7c0: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.004s
I20260812 07:04:38.073016 15911 raft_consensus.cc:348] T 35153c14a1e144d28ca5fa4e587a1ec5 P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:38.073320 15911 raft_consensus.cc:374] T 35153c14a1e144d28ca5fa4e587a1ec5 P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:38.073431 15911 raft_consensus.cc:729] T 35153c14a1e144d28ca5fa4e587a1ec5 P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59b3abeeb4124526b938e9dc4cf9a7c0, State: Initialized, Role: FOLLOWER
I20260812 07:04:38.073768 15911 consensus_queue.cc:260] T 35153c14a1e144d28ca5fa4e587a1ec5 P 59b3abeeb4124526b938e9dc4cf9a7c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:38.074383 15917 tablet_replica.cc:439] T 35153c14a1e144d28ca5fa4e587a1ec5 P 59b3abeeb4124526b938e9dc4cf9a7c0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } } }
I20260812 07:04:38.074554 15911 ts_tablet_manager.cc:1428] T 35153c14a1e144d28ca5fa4e587a1ec5 P 59b3abeeb4124526b938e9dc4cf9a7c0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 07:04:38.074703 15917 tablet_replica.cc:442] T 35153c14a1e144d28ca5fa4e587a1ec5 P 59b3abeeb4124526b938e9dc4cf9a7c0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:04:38.095203 15919 raft_consensus.cc:482] T 35153c14a1e144d28ca5fa4e587a1ec5 P e32364b73e254343b4bb6991207c2f85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:04:38.095422 15919 raft_consensus.cc:504] T 35153c14a1e144d28ca5fa4e587a1ec5 P e32364b73e254343b4bb6991207c2f85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:38.096094 15919 leader_election.cc:288] T 35153c14a1e144d28ca5fa4e587a1ec5 P e32364b73e254343b4bb6991207c2f85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2da9547064764ae9bfb18d37ce320d7c (127.6.61.1:34611), 59b3abeeb4124526b938e9dc4cf9a7c0 (127.6.61.3:33203)
I20260812 07:04:38.096719 15827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35153c14a1e144d28ca5fa4e587a1ec5" candidate_uuid: "e32364b73e254343b4bb6991207c2f85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" is_pre_election: true
I20260812 07:04:38.096678 15503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35153c14a1e144d28ca5fa4e587a1ec5" candidate_uuid: "e32364b73e254343b4bb6991207c2f85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2da9547064764ae9bfb18d37ce320d7c" is_pre_election: true
I20260812 07:04:38.096987 15827 raft_consensus.cc:2454] T 35153c14a1e144d28ca5fa4e587a1ec5 P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e32364b73e254343b4bb6991207c2f85 in term 0.
I20260812 07:04:38.096982 15503 raft_consensus.cc:2454] T 35153c14a1e144d28ca5fa4e587a1ec5 P 2da9547064764ae9bfb18d37ce320d7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e32364b73e254343b4bb6991207c2f85 in term 0.
I20260812 07:04:38.097451 15582 leader_election.cc:302] T 35153c14a1e144d28ca5fa4e587a1ec5 P e32364b73e254343b4bb6991207c2f85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2da9547064764ae9bfb18d37ce320d7c, e32364b73e254343b4bb6991207c2f85; no voters:
I20260812 07:04:38.097745 15919 raft_consensus.cc:2781] T 35153c14a1e144d28ca5fa4e587a1ec5 P e32364b73e254343b4bb6991207c2f85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:04:38.097868 15919 raft_consensus.cc:482] T 35153c14a1e144d28ca5fa4e587a1ec5 P e32364b73e254343b4bb6991207c2f85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:04:38.097997 15919 raft_consensus.cc:3037] T 35153c14a1e144d28ca5fa4e587a1ec5 P e32364b73e254343b4bb6991207c2f85 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:38.100054 15919 raft_consensus.cc:504] T 35153c14a1e144d28ca5fa4e587a1ec5 P e32364b73e254343b4bb6991207c2f85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:38.100703 15919 leader_election.cc:288] T 35153c14a1e144d28ca5fa4e587a1ec5 P e32364b73e254343b4bb6991207c2f85 [CANDIDATE]: Term 1 election: Requested vote from peers 2da9547064764ae9bfb18d37ce320d7c (127.6.61.1:34611), 59b3abeeb4124526b938e9dc4cf9a7c0 (127.6.61.3:33203)
I20260812 07:04:38.101133 15503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35153c14a1e144d28ca5fa4e587a1ec5" candidate_uuid: "e32364b73e254343b4bb6991207c2f85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2da9547064764ae9bfb18d37ce320d7c"
I20260812 07:04:38.101397 15503 raft_consensus.cc:3037] T 35153c14a1e144d28ca5fa4e587a1ec5 P 2da9547064764ae9bfb18d37ce320d7c [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:38.101490 15827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35153c14a1e144d28ca5fa4e587a1ec5" candidate_uuid: "e32364b73e254343b4bb6991207c2f85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0"
I20260812 07:04:38.101794 15827 raft_consensus.cc:3037] T 35153c14a1e144d28ca5fa4e587a1ec5 P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:38.103595 15503 raft_consensus.cc:2454] T 35153c14a1e144d28ca5fa4e587a1ec5 P 2da9547064764ae9bfb18d37ce320d7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e32364b73e254343b4bb6991207c2f85 in term 1.
I20260812 07:04:38.104209 15582 leader_election.cc:302] T 35153c14a1e144d28ca5fa4e587a1ec5 P e32364b73e254343b4bb6991207c2f85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2da9547064764ae9bfb18d37ce320d7c, e32364b73e254343b4bb6991207c2f85; no voters:
I20260812 07:04:38.104525 15919 raft_consensus.cc:2781] T 35153c14a1e144d28ca5fa4e587a1ec5 P e32364b73e254343b4bb6991207c2f85 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:04:38.104705 15919 raft_consensus.cc:686] T 35153c14a1e144d28ca5fa4e587a1ec5 P e32364b73e254343b4bb6991207c2f85 [term 1 LEADER]: Becoming Leader. State: Replica: e32364b73e254343b4bb6991207c2f85, State: Running, Role: LEADER
I20260812 07:04:38.104954 15919 consensus_queue.cc:237] T 35153c14a1e144d28ca5fa4e587a1ec5 P e32364b73e254343b4bb6991207c2f85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:38.105407 15827 raft_consensus.cc:2454] T 35153c14a1e144d28ca5fa4e587a1ec5 P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e32364b73e254343b4bb6991207c2f85 in term 1.
I20260812 07:04:38.106829 15946 tablet_replica.cc:439] T 35153c14a1e144d28ca5fa4e587a1ec5 P e32364b73e254343b4bb6991207c2f85: TxnStatusTablet state changed. Reason: New leader e32364b73e254343b4bb6991207c2f85. Latest consensus state: current_term: 1 leader_uuid: "e32364b73e254343b4bb6991207c2f85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } } }
I20260812 07:04:38.107134 15946 tablet_replica.cc:442] T 35153c14a1e144d28ca5fa4e587a1ec5 P e32364b73e254343b4bb6991207c2f85: This TxnStatusTablet replica's current role is: LEADER
I20260812 07:04:38.107945 15962 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 07:04:38.108247 15962 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 07:04:38.108947 15324 catalog_manager.cc:5495] T 35153c14a1e144d28ca5fa4e587a1ec5 P e32364b73e254343b4bb6991207c2f85 reported cstate change: term changed from 0 to 1, leader changed from <none> to e32364b73e254343b4bb6991207c2f85 (127.6.61.2). New cstate: current_term: 1 leader_uuid: "e32364b73e254343b4bb6991207c2f85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } health_report { overall_health: UNKNOWN } } }
I20260812 07:04:38.136000 15957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:38.127280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:35663 (local address 127.0.0.1:33458)
0812 07:04:38.127534 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:38.127558 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:38.127729 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:38.128416 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:38.128423 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:38.128436 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 07:04:38.128674 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:38.128687 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:38.130621 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:38.130628 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:38.132383 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:38.132401 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:38.133563 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:38.133570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:38.133794 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:38.133916 (+ 122us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 07:04:38.135313 (+ 1397us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 07:04:38.135343 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 07:04:38.135446 (+ 103us) client_negotiation.cc:240] Negotiation successful
0812 07:04:38.135638 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260812 07:04:38.136072 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:38.126803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33458 (local address 127.6.61.2:35663)
0812 07:04:38.127578 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:38.127585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:38.127634 (+ 49us) server_negotiation.cc:407] Connection header received
0812 07:04:38.128017 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:38.128027 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:38.128169 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:38.128380 (+ 211us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:04:38.128876 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:38.130422 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:38.132565 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:38.133363 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:38.133528 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:38.134216 (+ 688us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:04:38.135164 (+ 948us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:04:38.135335 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:38.135520 (+ 185us) server_negotiation.cc:299] Negotiation successful
0812 07:04:38.135638 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":352,"threads_started":1}
I20260812 07:04:38.141680 15827 raft_consensus.cc:1261] T 35153c14a1e144d28ca5fa4e587a1ec5 P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 1 FOLLOWER]: Refusing update from remote peer e32364b73e254343b4bb6991207c2f85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:04:38.142625 15503 raft_consensus.cc:1261] T 35153c14a1e144d28ca5fa4e587a1ec5 P 2da9547064764ae9bfb18d37ce320d7c [term 1 FOLLOWER]: Refusing update from remote peer e32364b73e254343b4bb6991207c2f85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:04:38.143028 15946 consensus_queue.cc:1035] T 35153c14a1e144d28ca5fa4e587a1ec5 P e32364b73e254343b4bb6991207c2f85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:04:38.143431 15919 consensus_queue.cc:1035] T 35153c14a1e144d28ca5fa4e587a1ec5 P e32364b73e254343b4bb6991207c2f85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:04:38.148381 15918 tablet_replica.cc:439] T 35153c14a1e144d28ca5fa4e587a1ec5 P 2da9547064764ae9bfb18d37ce320d7c: TxnStatusTablet state changed. Reason: New leader e32364b73e254343b4bb6991207c2f85. Latest consensus state: current_term: 1 leader_uuid: "e32364b73e254343b4bb6991207c2f85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } } }
I20260812 07:04:38.148737 15918 tablet_replica.cc:442] T 35153c14a1e144d28ca5fa4e587a1ec5 P 2da9547064764ae9bfb18d37ce320d7c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:04:38.153667 15929 mvcc.cc:205] Tried to move back new op lower bound from 7317578867258707968 to 7317578867120488448. Current Snapshot: MvccSnapshot[applied={T|T < 7317578867258707968}]
I20260812 07:04:38.154608 15918 tablet_replica.cc:439] T 35153c14a1e144d28ca5fa4e587a1ec5 P 2da9547064764ae9bfb18d37ce320d7c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e32364b73e254343b4bb6991207c2f85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } } }
I20260812 07:04:38.154850 15918 tablet_replica.cc:442] T 35153c14a1e144d28ca5fa4e587a1ec5 P 2da9547064764ae9bfb18d37ce320d7c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:04:38.156066 15919 tablet_replica.cc:439] T 35153c14a1e144d28ca5fa4e587a1ec5 P e32364b73e254343b4bb6991207c2f85: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e32364b73e254343b4bb6991207c2f85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } } }
I20260812 07:04:38.156390 15919 tablet_replica.cc:442] T 35153c14a1e144d28ca5fa4e587a1ec5 P e32364b73e254343b4bb6991207c2f85: This TxnStatusTablet replica's current role is: LEADER
I20260812 07:04:38.169186 15917 tablet_replica.cc:439] T 35153c14a1e144d28ca5fa4e587a1ec5 P 59b3abeeb4124526b938e9dc4cf9a7c0: TxnStatusTablet state changed. Reason: New leader e32364b73e254343b4bb6991207c2f85. Latest consensus state: current_term: 1 leader_uuid: "e32364b73e254343b4bb6991207c2f85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } } }
I20260812 07:04:38.169601 15917 tablet_replica.cc:442] T 35153c14a1e144d28ca5fa4e587a1ec5 P 59b3abeeb4124526b938e9dc4cf9a7c0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:04:38.172977 15946 tablet_replica.cc:439] T 35153c14a1e144d28ca5fa4e587a1ec5 P e32364b73e254343b4bb6991207c2f85: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e32364b73e254343b4bb6991207c2f85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } } }
I20260812 07:04:38.173272 15946 tablet_replica.cc:442] T 35153c14a1e144d28ca5fa4e587a1ec5 P e32364b73e254343b4bb6991207c2f85: This TxnStatusTablet replica's current role is: LEADER
I20260812 07:04:38.175196 15926 mvcc.cc:205] Tried to move back new op lower bound from 7317578867258707968 to 7317578867120488448. Current Snapshot: MvccSnapshot[applied={T|T < 7317578867258707968}]
I20260812 07:04:38.182405 15917 tablet_replica.cc:439] T 35153c14a1e144d28ca5fa4e587a1ec5 P 59b3abeeb4124526b938e9dc4cf9a7c0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e32364b73e254343b4bb6991207c2f85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } } }
I20260812 07:04:38.182729 15917 tablet_replica.cc:442] T 35153c14a1e144d28ca5fa4e587a1ec5 P 59b3abeeb4124526b938e9dc4cf9a7c0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:04:38.184410 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:38.168452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33464 (local address 127.6.61.2:35663)
0812 07:04:38.168655 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:38.168662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:38.177029 (+ 8367us) server_negotiation.cc:407] Connection header received
0812 07:04:38.177520 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:38.177526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:38.177629 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:38.177697 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:04:38.178477 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:38.179691 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:38.181984 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:38.182615 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:38.182722 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:38.183509 (+ 787us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:04:38.183676 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:04:38.183757 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:38.183993 (+ 236us) server_negotiation.cc:299] Negotiation successful
0812 07:04:38.184097 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 07:04:38.234473 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:38.207967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:57847 (local address 127.6.61.2:35663)
0812 07:04:38.208220 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:38.208226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:38.216388 (+ 8162us) server_negotiation.cc:407] Connection header received
0812 07:04:38.216872 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:38.216882 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:38.216954 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:38.217020 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:04:38.217802 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:38.219166 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:38.221577 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:38.230055 (+ 8478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:38.230191 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:38.233522 (+ 3331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:04:38.233653 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:04:38.233772 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:38.233988 (+ 216us) server_negotiation.cc:299] Negotiation successful
0812 07:04:38.234068 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 07:04:38.234584 15977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:38.208364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:35663 (local address 127.6.61.2:57847)
0812 07:04:38.216252 (+ 7888us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:38.216277 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:38.216724 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:38.217215 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:38.217220 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:38.217232 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 07:04:38.217629 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:38.217643 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:38.219328 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:38.219334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:38.221427 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:38.221444 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:38.230398 (+ 8954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:38.230404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:38.230615 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:38.230708 (+ 93us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 07:04:38.233791 (+ 3083us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 07:04:38.233806 (+ 15us) client_negotiation.cc:769] Sending connection context
0812 07:04:38.233953 (+ 147us) client_negotiation.cc:240] Negotiation successful
0812 07:04:38.234120 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7595,"thread_start_us":8042,"threads_started":1}
I20260812 07:04:38.246680 15827 raft_consensus.cc:1261] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 1 FOLLOWER]: Refusing update from remote peer e32364b73e254343b4bb6991207c2f85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:04:38.247561 15946 consensus_queue.cc:1035] T 176f91a5f60b45db872fbdfbb0c3fcb3 P e32364b73e254343b4bb6991207c2f85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:04:38.249447 15503 raft_consensus.cc:1261] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 2da9547064764ae9bfb18d37ce320d7c [term 1 FOLLOWER]: Refusing update from remote peer e32364b73e254343b4bb6991207c2f85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:04:38.250526 15919 consensus_queue.cc:1035] T 176f91a5f60b45db872fbdfbb0c3fcb3 P e32364b73e254343b4bb6991207c2f85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:04:38.266853 15927 mvcc.cc:205] Tried to move back new op lower bound from 7317578867691978752 to 7317578866353618944. Current Snapshot: MvccSnapshot[applied={T|T < 7317578867691978752}]
I20260812 07:04:38.342352 15324 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:04:38.342617 15324 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:04:38.345849 15324 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc0633cfe3c74aa6a8a6d4fbe3cec982: Sending DeleteTablet for 3 replicas of tablet 176f91a5f60b45db872fbdfbb0c3fcb3
I20260812 07:04:38.346728 15643 tablet_service.cc:1514] Processing DeleteTablet for tablet 176f91a5f60b45db872fbdfbb0c3fcb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:04:38 UTC) from {username='oryx', principal='oryx/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:33444
I20260812 07:04:38.346712 15481 tablet_service.cc:1514] Processing DeleteTablet for tablet 176f91a5f60b45db872fbdfbb0c3fcb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:04:38 UTC) from {username='oryx', principal='oryx/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:60892
I20260812 07:04:38.347227 15795 tablet_service.cc:1514] Processing DeleteTablet for tablet 176f91a5f60b45db872fbdfbb0c3fcb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:04:38 UTC) from {username='oryx', principal='oryx/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:51874
I20260812 07:04:38.347501 15982 tablet_replica.cc:330] T 176f91a5f60b45db872fbdfbb0c3fcb3 P e32364b73e254343b4bb6991207c2f85: stopping tablet replica
I20260812 07:04:38.347729 15982 raft_consensus.cc:2229] T 176f91a5f60b45db872fbdfbb0c3fcb3 P e32364b73e254343b4bb6991207c2f85 [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:04:38.348366 15982 raft_consensus.cc:2258] T 176f91a5f60b45db872fbdfbb0c3fcb3 P e32364b73e254343b4bb6991207c2f85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:04:38.348975 15983 tablet_replica.cc:330] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 2da9547064764ae9bfb18d37ce320d7c: stopping tablet replica
I20260812 07:04:38.349169 15983 raft_consensus.cc:2229] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 2da9547064764ae9bfb18d37ce320d7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:04:38.349402 15983 raft_consensus.cc:2258] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 2da9547064764ae9bfb18d37ce320d7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:04:38.350548 15983 ts_tablet_manager.cc:1905] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 2da9547064764ae9bfb18d37ce320d7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:04:38.353271 15982 ts_tablet_manager.cc:1905] T 176f91a5f60b45db872fbdfbb0c3fcb3 P e32364b73e254343b4bb6991207c2f85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:04:38.358690 15985 tablet_replica.cc:330] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 59b3abeeb4124526b938e9dc4cf9a7c0: stopping tablet replica
I20260812 07:04:38.359141 15985 raft_consensus.cc:2229] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:04:38.359519 15985 raft_consensus.cc:2258] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:04:38.360987 15985 ts_tablet_manager.cc:1905] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 59b3abeeb4124526b938e9dc4cf9a7c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:04:38.368386 15983 ts_tablet_manager.cc:1918] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 2da9547064764ae9bfb18d37ce320d7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 07:04:38.368583 15983 log.cc:1196] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 2da9547064764ae9bfb18d37ce320d7c: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-0/wal/wals/176f91a5f60b45db872fbdfbb0c3fcb3
I20260812 07:04:38.369141 15983 ts_tablet_manager.cc:1939] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 2da9547064764ae9bfb18d37ce320d7c: Deleting consensus metadata
I20260812 07:04:38.370178 15982 ts_tablet_manager.cc:1918] T 176f91a5f60b45db872fbdfbb0c3fcb3 P e32364b73e254343b4bb6991207c2f85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 07:04:38.370375 15982 log.cc:1196] T 176f91a5f60b45db872fbdfbb0c3fcb3 P e32364b73e254343b4bb6991207c2f85: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-1/wal/wals/176f91a5f60b45db872fbdfbb0c3fcb3
I20260812 07:04:38.370855 15985 ts_tablet_manager.cc:1918] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 59b3abeeb4124526b938e9dc4cf9a7c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 07:04:38.371001 15982 ts_tablet_manager.cc:1939] T 176f91a5f60b45db872fbdfbb0c3fcb3 P e32364b73e254343b4bb6991207c2f85: Deleting consensus metadata
I20260812 07:04:38.371116 15985 log.cc:1196] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 59b3abeeb4124526b938e9dc4cf9a7c0: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-2/wal/wals/176f91a5f60b45db872fbdfbb0c3fcb3
I20260812 07:04:38.371650 15985 ts_tablet_manager.cc:1939] T 176f91a5f60b45db872fbdfbb0c3fcb3 P 59b3abeeb4124526b938e9dc4cf9a7c0: Deleting consensus metadata
I20260812 07:04:38.372684 15305 catalog_manager.cc:4841] TS 2da9547064764ae9bfb18d37ce320d7c (127.6.61.1:34611): tablet 176f91a5f60b45db872fbdfbb0c3fcb3 (table test-table [id=dcd77504f7f342578e62b876ff9fc1c5]) successfully deleted
I20260812 07:04:38.372951 15307 catalog_manager.cc:4841] TS 59b3abeeb4124526b938e9dc4cf9a7c0 (127.6.61.3:33203): tablet 176f91a5f60b45db872fbdfbb0c3fcb3 (table test-table [id=dcd77504f7f342578e62b876ff9fc1c5]) successfully deleted
I20260812 07:04:38.373301 15309 catalog_manager.cc:4841] TS e32364b73e254343b4bb6991207c2f85 (127.6.61.2:35663): tablet 176f91a5f60b45db872fbdfbb0c3fcb3 (table test-table [id=dcd77504f7f342578e62b876ff9fc1c5]) successfully deleted
Aug 12 07:04:38 dist-test-slave-69rg krb5kdc[15268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 07:04:38 dist-test-slave-69rg krb5kdc[15268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:38 dist-test-slave-69rg krb5kdc[15268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 07:04:38.424655 15956 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:38.413205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50204 (local address 127.6.61.62:35059)
0812 07:04:38.413463 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:38.413472 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:38.413845 (+ 373us) server_negotiation.cc:407] Connection header received
0812 07:04:38.414057 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:38.414063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:38.414148 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:38.414266 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:38.415814 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:38.416917 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:38.418264 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:38.418882 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:38.419011 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:38.424360 (+ 5349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50204: BlockingRecv error: recv got EOF from 127.0.0.1:50204 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 12 07:04:38 dist-test-slave-69rg krb5kdc[15268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM
I20260812 07:04:38.444169 15956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:38.427807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50208 (local address 127.6.61.62:35059)
0812 07:04:38.428006 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:38.428014 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:38.428587 (+ 573us) server_negotiation.cc:407] Connection header received
0812 07:04:38.428776 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:38.428780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:38.428847 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:38.428930 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:38.430570 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:38.431354 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:38.433299 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:38.433720 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:38.433882 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:38.438615 (+ 4733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:38.438620 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:38.438623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:38.438684 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:38.441339 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:38.442076 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:38.442082 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:38.442087 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:38.442262 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:38.442682 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:38.442688 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:38.442691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:38.442973 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:38.443252 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:38.443558 (+ 306us) server_negotiation.cc:299] Negotiation successful
0812 07:04:38.443810 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 07:04:38.448207 15324 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50208:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:04:38.448987 15324 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:04:38.456574 15481 tablet_service.cc:1467] Processing CreateTablet for tablet f486c44ce52544a188e4caa0c991e7d0 (DEFAULT_TABLE table=test-table [id=7fcff413488c40c7887165ee51a5b957]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:04:38.457074 15481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f486c44ce52544a188e4caa0c991e7d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:38.457036 15643 tablet_service.cc:1467] Processing CreateTablet for tablet f486c44ce52544a188e4caa0c991e7d0 (DEFAULT_TABLE table=test-table [id=7fcff413488c40c7887165ee51a5b957]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:04:38.457480 15643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f486c44ce52544a188e4caa0c991e7d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:38.458561 15795 tablet_service.cc:1467] Processing CreateTablet for tablet f486c44ce52544a188e4caa0c991e7d0 (DEFAULT_TABLE table=test-table [id=7fcff413488c40c7887165ee51a5b957]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:04:38.459059 15795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f486c44ce52544a188e4caa0c991e7d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:38.463914 15910 tablet_bootstrap.cc:492] T f486c44ce52544a188e4caa0c991e7d0 P 2da9547064764ae9bfb18d37ce320d7c: Bootstrap starting.
I20260812 07:04:38.465137 15909 tablet_bootstrap.cc:492] T f486c44ce52544a188e4caa0c991e7d0 P e32364b73e254343b4bb6991207c2f85: Bootstrap starting.
I20260812 07:04:38.465611 15910 tablet_bootstrap.cc:654] T f486c44ce52544a188e4caa0c991e7d0 P 2da9547064764ae9bfb18d37ce320d7c: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:38.467156 15909 tablet_bootstrap.cc:654] T f486c44ce52544a188e4caa0c991e7d0 P e32364b73e254343b4bb6991207c2f85: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:38.468580 15911 tablet_bootstrap.cc:492] T f486c44ce52544a188e4caa0c991e7d0 P 59b3abeeb4124526b938e9dc4cf9a7c0: Bootstrap starting.
I20260812 07:04:38.470839 15911 tablet_bootstrap.cc:654] T f486c44ce52544a188e4caa0c991e7d0 P 59b3abeeb4124526b938e9dc4cf9a7c0: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:38.474258 15909 tablet_bootstrap.cc:492] T f486c44ce52544a188e4caa0c991e7d0 P e32364b73e254343b4bb6991207c2f85: No bootstrap required, opened a new log
I20260812 07:04:38.474550 15909 ts_tablet_manager.cc:1397] T f486c44ce52544a188e4caa0c991e7d0 P e32364b73e254343b4bb6991207c2f85: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.003s
I20260812 07:04:38.475715 15909 raft_consensus.cc:348] T f486c44ce52544a188e4caa0c991e7d0 P e32364b73e254343b4bb6991207c2f85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:38.476176 15909 raft_consensus.cc:374] T f486c44ce52544a188e4caa0c991e7d0 P e32364b73e254343b4bb6991207c2f85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:38.476310 15909 raft_consensus.cc:729] T f486c44ce52544a188e4caa0c991e7d0 P e32364b73e254343b4bb6991207c2f85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e32364b73e254343b4bb6991207c2f85, State: Initialized, Role: FOLLOWER
I20260812 07:04:38.476681 15909 consensus_queue.cc:260] T f486c44ce52544a188e4caa0c991e7d0 P e32364b73e254343b4bb6991207c2f85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:38.477654 15909 ts_tablet_manager.cc:1428] T f486c44ce52544a188e4caa0c991e7d0 P e32364b73e254343b4bb6991207c2f85: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260812 07:04:38.479095 15911 tablet_bootstrap.cc:492] T f486c44ce52544a188e4caa0c991e7d0 P 59b3abeeb4124526b938e9dc4cf9a7c0: No bootstrap required, opened a new log
I20260812 07:04:38.479285 15911 ts_tablet_manager.cc:1397] T f486c44ce52544a188e4caa0c991e7d0 P 59b3abeeb4124526b938e9dc4cf9a7c0: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260812 07:04:38.480268 15911 raft_consensus.cc:348] T f486c44ce52544a188e4caa0c991e7d0 P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:38.480680 15911 raft_consensus.cc:374] T f486c44ce52544a188e4caa0c991e7d0 P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:38.480798 15911 raft_consensus.cc:729] T f486c44ce52544a188e4caa0c991e7d0 P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59b3abeeb4124526b938e9dc4cf9a7c0, State: Initialized, Role: FOLLOWER
I20260812 07:04:38.481159 15911 consensus_queue.cc:260] T f486c44ce52544a188e4caa0c991e7d0 P 59b3abeeb4124526b938e9dc4cf9a7c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:38.483350 15911 ts_tablet_manager.cc:1428] T f486c44ce52544a188e4caa0c991e7d0 P 59b3abeeb4124526b938e9dc4cf9a7c0: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260812 07:04:38.485190 15910 tablet_bootstrap.cc:492] T f486c44ce52544a188e4caa0c991e7d0 P 2da9547064764ae9bfb18d37ce320d7c: No bootstrap required, opened a new log
I20260812 07:04:38.485314 15910 ts_tablet_manager.cc:1397] T f486c44ce52544a188e4caa0c991e7d0 P 2da9547064764ae9bfb18d37ce320d7c: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.004s
I20260812 07:04:38.486205 15910 raft_consensus.cc:348] T f486c44ce52544a188e4caa0c991e7d0 P 2da9547064764ae9bfb18d37ce320d7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:38.486438 15910 raft_consensus.cc:374] T f486c44ce52544a188e4caa0c991e7d0 P 2da9547064764ae9bfb18d37ce320d7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:38.486554 15910 raft_consensus.cc:729] T f486c44ce52544a188e4caa0c991e7d0 P 2da9547064764ae9bfb18d37ce320d7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2da9547064764ae9bfb18d37ce320d7c, State: Initialized, Role: FOLLOWER
I20260812 07:04:38.486809 15910 consensus_queue.cc:260] T f486c44ce52544a188e4caa0c991e7d0 P 2da9547064764ae9bfb18d37ce320d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:38.487329 15910 ts_tablet_manager.cc:1428] T f486c44ce52544a188e4caa0c991e7d0 P 2da9547064764ae9bfb18d37ce320d7c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 07:04:38.678758 15917 raft_consensus.cc:482] T f486c44ce52544a188e4caa0c991e7d0 P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:04:38.678988 15917 raft_consensus.cc:504] T f486c44ce52544a188e4caa0c991e7d0 P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:38.679788 15917 leader_election.cc:288] T f486c44ce52544a188e4caa0c991e7d0 P 59b3abeeb4124526b938e9dc4cf9a7c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2da9547064764ae9bfb18d37ce320d7c (127.6.61.1:34611), e32364b73e254343b4bb6991207c2f85 (127.6.61.2:35663)
I20260812 07:04:38.680424 15665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f486c44ce52544a188e4caa0c991e7d0" candidate_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e32364b73e254343b4bb6991207c2f85" is_pre_election: true
I20260812 07:04:38.680441 15503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f486c44ce52544a188e4caa0c991e7d0" candidate_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2da9547064764ae9bfb18d37ce320d7c" is_pre_election: true
I20260812 07:04:38.680711 15503 raft_consensus.cc:2454] T f486c44ce52544a188e4caa0c991e7d0 P 2da9547064764ae9bfb18d37ce320d7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59b3abeeb4124526b938e9dc4cf9a7c0 in term 0.
I20260812 07:04:38.680706 15665 raft_consensus.cc:2454] T f486c44ce52544a188e4caa0c991e7d0 P e32364b73e254343b4bb6991207c2f85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59b3abeeb4124526b938e9dc4cf9a7c0 in term 0.
I20260812 07:04:38.681320 15739 leader_election.cc:302] T f486c44ce52544a188e4caa0c991e7d0 P 59b3abeeb4124526b938e9dc4cf9a7c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2da9547064764ae9bfb18d37ce320d7c, 59b3abeeb4124526b938e9dc4cf9a7c0; no voters:
I20260812 07:04:38.681639 15917 raft_consensus.cc:2781] T f486c44ce52544a188e4caa0c991e7d0 P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:04:38.681787 15917 raft_consensus.cc:482] T f486c44ce52544a188e4caa0c991e7d0 P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:04:38.681968 15917 raft_consensus.cc:3037] T f486c44ce52544a188e4caa0c991e7d0 P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:38.684087 15917 raft_consensus.cc:504] T f486c44ce52544a188e4caa0c991e7d0 P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:38.684670 15917 leader_election.cc:288] T f486c44ce52544a188e4caa0c991e7d0 P 59b3abeeb4124526b938e9dc4cf9a7c0 [CANDIDATE]: Term 1 election: Requested vote from peers 2da9547064764ae9bfb18d37ce320d7c (127.6.61.1:34611), e32364b73e254343b4bb6991207c2f85 (127.6.61.2:35663)
I20260812 07:04:38.685205 15503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f486c44ce52544a188e4caa0c991e7d0" candidate_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2da9547064764ae9bfb18d37ce320d7c"
I20260812 07:04:38.685246 15665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f486c44ce52544a188e4caa0c991e7d0" candidate_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e32364b73e254343b4bb6991207c2f85"
I20260812 07:04:38.685436 15503 raft_consensus.cc:3037] T f486c44ce52544a188e4caa0c991e7d0 P 2da9547064764ae9bfb18d37ce320d7c [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:38.685453 15665 raft_consensus.cc:3037] T f486c44ce52544a188e4caa0c991e7d0 P e32364b73e254343b4bb6991207c2f85 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:38.687527 15503 raft_consensus.cc:2454] T f486c44ce52544a188e4caa0c991e7d0 P 2da9547064764ae9bfb18d37ce320d7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59b3abeeb4124526b938e9dc4cf9a7c0 in term 1.
I20260812 07:04:38.687953 15739 leader_election.cc:302] T f486c44ce52544a188e4caa0c991e7d0 P 59b3abeeb4124526b938e9dc4cf9a7c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2da9547064764ae9bfb18d37ce320d7c, 59b3abeeb4124526b938e9dc4cf9a7c0; no voters:
I20260812 07:04:38.688254 15917 raft_consensus.cc:2781] T f486c44ce52544a188e4caa0c991e7d0 P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:04:38.688669 15665 raft_consensus.cc:2454] T f486c44ce52544a188e4caa0c991e7d0 P e32364b73e254343b4bb6991207c2f85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59b3abeeb4124526b938e9dc4cf9a7c0 in term 1.
I20260812 07:04:38.688726 15917 raft_consensus.cc:686] T f486c44ce52544a188e4caa0c991e7d0 P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 1 LEADER]: Becoming Leader. State: Replica: 59b3abeeb4124526b938e9dc4cf9a7c0, State: Running, Role: LEADER
I20260812 07:04:38.689291 15917 consensus_queue.cc:237] T f486c44ce52544a188e4caa0c991e7d0 P 59b3abeeb4124526b938e9dc4cf9a7c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:38.692199 15326 catalog_manager.cc:5495] T f486c44ce52544a188e4caa0c991e7d0 P 59b3abeeb4124526b938e9dc4cf9a7c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59b3abeeb4124526b938e9dc4cf9a7c0 (127.6.61.3). New cstate: current_term: 1 leader_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } health_report { overall_health: HEALTHY } } }
I20260812 07:04:38.719969 16007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:38.713757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51890 (local address 127.6.61.3:33203)
0812 07:04:38.714455 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:38.714460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:38.714478 (+ 18us) server_negotiation.cc:407] Connection header received
0812 07:04:38.714539 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:38.714543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:38.714609 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:38.714695 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:04:38.715175 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:38.716041 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:38.718070 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:38.718538 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:38.718644 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:38.719182 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:04:38.719296 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:04:38.719371 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:38.719544 (+ 173us) server_negotiation.cc:299] Negotiation successful
0812 07:04:38.719627 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":415,"threads_started":1}
I20260812 07:04:38.723191 15503 raft_consensus.cc:1261] T f486c44ce52544a188e4caa0c991e7d0 P 2da9547064764ae9bfb18d37ce320d7c [term 1 FOLLOWER]: Refusing update from remote peer 59b3abeeb4124526b938e9dc4cf9a7c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:04:38.723181 15665 raft_consensus.cc:1261] T f486c44ce52544a188e4caa0c991e7d0 P e32364b73e254343b4bb6991207c2f85 [term 1 FOLLOWER]: Refusing update from remote peer 59b3abeeb4124526b938e9dc4cf9a7c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:04:38.723898 15917 consensus_queue.cc:1035] T f486c44ce52544a188e4caa0c991e7d0 P 59b3abeeb4124526b938e9dc4cf9a7c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:04:38.724223 16003 consensus_queue.cc:1035] T f486c44ce52544a188e4caa0c991e7d0 P 59b3abeeb4124526b938e9dc4cf9a7c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:04:38.738667 15326 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:04:38.739040 15326 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:04:38.742170 15326 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc0633cfe3c74aa6a8a6d4fbe3cec982: Sending DeleteTablet for 3 replicas of tablet f486c44ce52544a188e4caa0c991e7d0
I20260812 07:04:38.742971 15481 tablet_service.cc:1514] Processing DeleteTablet for tablet f486c44ce52544a188e4caa0c991e7d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:04:38 UTC) from {username='oryx', principal='oryx/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:60892
I20260812 07:04:38.743053 15643 tablet_service.cc:1514] Processing DeleteTablet for tablet f486c44ce52544a188e4caa0c991e7d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:04:38 UTC) from {username='oryx', principal='oryx/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:33444
I20260812 07:04:38.743270 15983 tablet_replica.cc:330] T f486c44ce52544a188e4caa0c991e7d0 P 2da9547064764ae9bfb18d37ce320d7c: stopping tablet replica
I20260812 07:04:38.743348 15982 tablet_replica.cc:330] T f486c44ce52544a188e4caa0c991e7d0 P e32364b73e254343b4bb6991207c2f85: stopping tablet replica
I20260812 07:04:38.743312 15795 tablet_service.cc:1514] Processing DeleteTablet for tablet f486c44ce52544a188e4caa0c991e7d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:04:38 UTC) from {username='oryx', principal='oryx/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:51874
I20260812 07:04:38.743465 15983 raft_consensus.cc:2229] T f486c44ce52544a188e4caa0c991e7d0 P 2da9547064764ae9bfb18d37ce320d7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:04:38.743547 15982 raft_consensus.cc:2229] T f486c44ce52544a188e4caa0c991e7d0 P e32364b73e254343b4bb6991207c2f85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:04:38.743623 15985 tablet_replica.cc:330] T f486c44ce52544a188e4caa0c991e7d0 P 59b3abeeb4124526b938e9dc4cf9a7c0: stopping tablet replica
I20260812 07:04:38.743686 15983 raft_consensus.cc:2258] T f486c44ce52544a188e4caa0c991e7d0 P 2da9547064764ae9bfb18d37ce320d7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:04:38.743798 15982 raft_consensus.cc:2258] T f486c44ce52544a188e4caa0c991e7d0 P e32364b73e254343b4bb6991207c2f85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:04:38.743804 15985 raft_consensus.cc:2229] T f486c44ce52544a188e4caa0c991e7d0 P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:04:38.744123 15985 raft_consensus.cc:2258] T f486c44ce52544a188e4caa0c991e7d0 P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:04:38.744874 15982 ts_tablet_manager.cc:1905] T f486c44ce52544a188e4caa0c991e7d0 P e32364b73e254343b4bb6991207c2f85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:04:38.745677 15985 ts_tablet_manager.cc:1905] T f486c44ce52544a188e4caa0c991e7d0 P 59b3abeeb4124526b938e9dc4cf9a7c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:04:38.745644 15326 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50208:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:04:38.746325 15326 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:04:38.750507 15983 ts_tablet_manager.cc:1905] T f486c44ce52544a188e4caa0c991e7d0 P 2da9547064764ae9bfb18d37ce320d7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:04:38.751525 15985 ts_tablet_manager.cc:1918] T f486c44ce52544a188e4caa0c991e7d0 P 59b3abeeb4124526b938e9dc4cf9a7c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:04:38.751756 15985 log.cc:1196] T f486c44ce52544a188e4caa0c991e7d0 P 59b3abeeb4124526b938e9dc4cf9a7c0: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-2/wal/wals/f486c44ce52544a188e4caa0c991e7d0
I20260812 07:04:38.752146 15982 ts_tablet_manager.cc:1918] T f486c44ce52544a188e4caa0c991e7d0 P e32364b73e254343b4bb6991207c2f85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:04:38.752322 15982 log.cc:1196] T f486c44ce52544a188e4caa0c991e7d0 P e32364b73e254343b4bb6991207c2f85: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-1/wal/wals/f486c44ce52544a188e4caa0c991e7d0
I20260812 07:04:38.752444 15985 ts_tablet_manager.cc:1939] T f486c44ce52544a188e4caa0c991e7d0 P 59b3abeeb4124526b938e9dc4cf9a7c0: Deleting consensus metadata
I20260812 07:04:38.752837 15982 ts_tablet_manager.cc:1939] T f486c44ce52544a188e4caa0c991e7d0 P e32364b73e254343b4bb6991207c2f85: Deleting consensus metadata
I20260812 07:04:38.753837 15307 catalog_manager.cc:4841] TS 59b3abeeb4124526b938e9dc4cf9a7c0 (127.6.61.3:33203): tablet f486c44ce52544a188e4caa0c991e7d0 (table test-table [id=7fcff413488c40c7887165ee51a5b957]) successfully deleted
I20260812 07:04:38.755064 15309 catalog_manager.cc:4841] TS e32364b73e254343b4bb6991207c2f85 (127.6.61.2:35663): tablet f486c44ce52544a188e4caa0c991e7d0 (table test-table [id=7fcff413488c40c7887165ee51a5b957]) successfully deleted
I20260812 07:04:38.758265 15983 ts_tablet_manager.cc:1918] T f486c44ce52544a188e4caa0c991e7d0 P 2da9547064764ae9bfb18d37ce320d7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:04:38.758440 15983 log.cc:1196] T f486c44ce52544a188e4caa0c991e7d0 P 2da9547064764ae9bfb18d37ce320d7c: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-0/wal/wals/f486c44ce52544a188e4caa0c991e7d0
I20260812 07:04:38.758962 15983 ts_tablet_manager.cc:1939] T f486c44ce52544a188e4caa0c991e7d0 P 2da9547064764ae9bfb18d37ce320d7c: Deleting consensus metadata
I20260812 07:04:38.759768 15643 tablet_service.cc:1467] Processing CreateTablet for tablet a3fe12b664a9457ea5f1c367d2cd746c (DEFAULT_TABLE table=test-table [id=9fb27b86b0004e228530c9e809049174]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:04:38.759888 15481 tablet_service.cc:1467] Processing CreateTablet for tablet a3fe12b664a9457ea5f1c367d2cd746c (DEFAULT_TABLE table=test-table [id=9fb27b86b0004e228530c9e809049174]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:04:38.760243 15643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3fe12b664a9457ea5f1c367d2cd746c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:38.760334 15481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3fe12b664a9457ea5f1c367d2cd746c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:38.761318 15795 tablet_service.cc:1467] Processing CreateTablet for tablet a3fe12b664a9457ea5f1c367d2cd746c (DEFAULT_TABLE table=test-table [id=9fb27b86b0004e228530c9e809049174]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:04:38.761734 15795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3fe12b664a9457ea5f1c367d2cd746c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:38.765753 15305 catalog_manager.cc:4841] TS 2da9547064764ae9bfb18d37ce320d7c (127.6.61.1:34611): tablet f486c44ce52544a188e4caa0c991e7d0 (table test-table [id=7fcff413488c40c7887165ee51a5b957]) successfully deleted
I20260812 07:04:38.766906 15910 tablet_bootstrap.cc:492] T a3fe12b664a9457ea5f1c367d2cd746c P 2da9547064764ae9bfb18d37ce320d7c: Bootstrap starting.
I20260812 07:04:38.767576 15909 tablet_bootstrap.cc:492] T a3fe12b664a9457ea5f1c367d2cd746c P e32364b73e254343b4bb6991207c2f85: Bootstrap starting.
I20260812 07:04:38.768568 15910 tablet_bootstrap.cc:654] T a3fe12b664a9457ea5f1c367d2cd746c P 2da9547064764ae9bfb18d37ce320d7c: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:38.769294 15909 tablet_bootstrap.cc:654] T a3fe12b664a9457ea5f1c367d2cd746c P e32364b73e254343b4bb6991207c2f85: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:38.769682 15911 tablet_bootstrap.cc:492] T a3fe12b664a9457ea5f1c367d2cd746c P 59b3abeeb4124526b938e9dc4cf9a7c0: Bootstrap starting.
I20260812 07:04:38.771854 15911 tablet_bootstrap.cc:654] T a3fe12b664a9457ea5f1c367d2cd746c P 59b3abeeb4124526b938e9dc4cf9a7c0: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:38.780772 15911 tablet_bootstrap.cc:492] T a3fe12b664a9457ea5f1c367d2cd746c P 59b3abeeb4124526b938e9dc4cf9a7c0: No bootstrap required, opened a new log
I20260812 07:04:38.780972 15911 ts_tablet_manager.cc:1397] T a3fe12b664a9457ea5f1c367d2cd746c P 59b3abeeb4124526b938e9dc4cf9a7c0: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260812 07:04:38.782024 15911 raft_consensus.cc:348] T a3fe12b664a9457ea5f1c367d2cd746c P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:38.782371 15911 raft_consensus.cc:374] T a3fe12b664a9457ea5f1c367d2cd746c P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:38.782464 15911 raft_consensus.cc:729] T a3fe12b664a9457ea5f1c367d2cd746c P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59b3abeeb4124526b938e9dc4cf9a7c0, State: Initialized, Role: FOLLOWER
I20260812 07:04:38.782783 15911 consensus_queue.cc:260] T a3fe12b664a9457ea5f1c367d2cd746c P 59b3abeeb4124526b938e9dc4cf9a7c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:38.783393 15911 ts_tablet_manager.cc:1428] T a3fe12b664a9457ea5f1c367d2cd746c P 59b3abeeb4124526b938e9dc4cf9a7c0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 07:04:38.784163 15910 tablet_bootstrap.cc:492] T a3fe12b664a9457ea5f1c367d2cd746c P 2da9547064764ae9bfb18d37ce320d7c: No bootstrap required, opened a new log
I20260812 07:04:38.784404 15910 ts_tablet_manager.cc:1397] T a3fe12b664a9457ea5f1c367d2cd746c P 2da9547064764ae9bfb18d37ce320d7c: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.005s
I20260812 07:04:38.785131 15909 tablet_bootstrap.cc:492] T a3fe12b664a9457ea5f1c367d2cd746c P e32364b73e254343b4bb6991207c2f85: No bootstrap required, opened a new log
I20260812 07:04:38.785320 15909 ts_tablet_manager.cc:1397] T a3fe12b664a9457ea5f1c367d2cd746c P e32364b73e254343b4bb6991207c2f85: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.010s
I20260812 07:04:38.786181 15909 raft_consensus.cc:348] T a3fe12b664a9457ea5f1c367d2cd746c P e32364b73e254343b4bb6991207c2f85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:38.786469 15909 raft_consensus.cc:374] T a3fe12b664a9457ea5f1c367d2cd746c P e32364b73e254343b4bb6991207c2f85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:38.786553 15909 raft_consensus.cc:729] T a3fe12b664a9457ea5f1c367d2cd746c P e32364b73e254343b4bb6991207c2f85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e32364b73e254343b4bb6991207c2f85, State: Initialized, Role: FOLLOWER
I20260812 07:04:38.786484 15910 raft_consensus.cc:348] T a3fe12b664a9457ea5f1c367d2cd746c P 2da9547064764ae9bfb18d37ce320d7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:38.786933 15910 raft_consensus.cc:374] T a3fe12b664a9457ea5f1c367d2cd746c P 2da9547064764ae9bfb18d37ce320d7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:38.786850 15909 consensus_queue.cc:260] T a3fe12b664a9457ea5f1c367d2cd746c P e32364b73e254343b4bb6991207c2f85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:38.787109 15910 raft_consensus.cc:729] T a3fe12b664a9457ea5f1c367d2cd746c P 2da9547064764ae9bfb18d37ce320d7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2da9547064764ae9bfb18d37ce320d7c, State: Initialized, Role: FOLLOWER
I20260812 07:04:38.787456 15910 consensus_queue.cc:260] T a3fe12b664a9457ea5f1c367d2cd746c P 2da9547064764ae9bfb18d37ce320d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:38.788167 15910 ts_tablet_manager.cc:1428] T a3fe12b664a9457ea5f1c367d2cd746c P 2da9547064764ae9bfb18d37ce320d7c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 07:04:38.788283 15909 ts_tablet_manager.cc:1428] T a3fe12b664a9457ea5f1c367d2cd746c P e32364b73e254343b4bb6991207c2f85: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 07:04:38.840346 15919 raft_consensus.cc:482] T a3fe12b664a9457ea5f1c367d2cd746c P e32364b73e254343b4bb6991207c2f85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:04:38.840574 15919 raft_consensus.cc:504] T a3fe12b664a9457ea5f1c367d2cd746c P e32364b73e254343b4bb6991207c2f85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:38.841394 15919 leader_election.cc:288] T a3fe12b664a9457ea5f1c367d2cd746c P e32364b73e254343b4bb6991207c2f85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2da9547064764ae9bfb18d37ce320d7c (127.6.61.1:34611), 59b3abeeb4124526b938e9dc4cf9a7c0 (127.6.61.3:33203)
I20260812 07:04:38.841989 15503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3fe12b664a9457ea5f1c367d2cd746c" candidate_uuid: "e32364b73e254343b4bb6991207c2f85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2da9547064764ae9bfb18d37ce320d7c" is_pre_election: true
I20260812 07:04:38.842211 15827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3fe12b664a9457ea5f1c367d2cd746c" candidate_uuid: "e32364b73e254343b4bb6991207c2f85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" is_pre_election: true
I20260812 07:04:38.842274 15503 raft_consensus.cc:2454] T a3fe12b664a9457ea5f1c367d2cd746c P 2da9547064764ae9bfb18d37ce320d7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e32364b73e254343b4bb6991207c2f85 in term 0.
I20260812 07:04:38.842410 15827 raft_consensus.cc:2454] T a3fe12b664a9457ea5f1c367d2cd746c P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e32364b73e254343b4bb6991207c2f85 in term 0.
I20260812 07:04:38.842831 15582 leader_election.cc:302] T a3fe12b664a9457ea5f1c367d2cd746c P e32364b73e254343b4bb6991207c2f85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2da9547064764ae9bfb18d37ce320d7c, e32364b73e254343b4bb6991207c2f85; no voters:
I20260812 07:04:38.843225 15919 raft_consensus.cc:2781] T a3fe12b664a9457ea5f1c367d2cd746c P e32364b73e254343b4bb6991207c2f85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:04:38.843391 15919 raft_consensus.cc:482] T a3fe12b664a9457ea5f1c367d2cd746c P e32364b73e254343b4bb6991207c2f85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:04:38.843525 15919 raft_consensus.cc:3037] T a3fe12b664a9457ea5f1c367d2cd746c P e32364b73e254343b4bb6991207c2f85 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:38.845923 15919 raft_consensus.cc:504] T a3fe12b664a9457ea5f1c367d2cd746c P e32364b73e254343b4bb6991207c2f85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:38.846534 15919 leader_election.cc:288] T a3fe12b664a9457ea5f1c367d2cd746c P e32364b73e254343b4bb6991207c2f85 [CANDIDATE]: Term 1 election: Requested vote from peers 2da9547064764ae9bfb18d37ce320d7c (127.6.61.1:34611), 59b3abeeb4124526b938e9dc4cf9a7c0 (127.6.61.3:33203)
I20260812 07:04:38.847084 15827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3fe12b664a9457ea5f1c367d2cd746c" candidate_uuid: "e32364b73e254343b4bb6991207c2f85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0"
I20260812 07:04:38.847074 15503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3fe12b664a9457ea5f1c367d2cd746c" candidate_uuid: "e32364b73e254343b4bb6991207c2f85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2da9547064764ae9bfb18d37ce320d7c"
I20260812 07:04:38.847314 15503 raft_consensus.cc:3037] T a3fe12b664a9457ea5f1c367d2cd746c P 2da9547064764ae9bfb18d37ce320d7c [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:38.847306 15827 raft_consensus.cc:3037] T a3fe12b664a9457ea5f1c367d2cd746c P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:38.849426 15827 raft_consensus.cc:2454] T a3fe12b664a9457ea5f1c367d2cd746c P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e32364b73e254343b4bb6991207c2f85 in term 1.
I20260812 07:04:38.849886 15584 leader_election.cc:302] T a3fe12b664a9457ea5f1c367d2cd746c P e32364b73e254343b4bb6991207c2f85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59b3abeeb4124526b938e9dc4cf9a7c0, e32364b73e254343b4bb6991207c2f85; no voters:
I20260812 07:04:38.850211 15919 raft_consensus.cc:2781] T a3fe12b664a9457ea5f1c367d2cd746c P e32364b73e254343b4bb6991207c2f85 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:04:38.850577 15503 raft_consensus.cc:2454] T a3fe12b664a9457ea5f1c367d2cd746c P 2da9547064764ae9bfb18d37ce320d7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e32364b73e254343b4bb6991207c2f85 in term 1.
I20260812 07:04:38.850821 15919 raft_consensus.cc:686] T a3fe12b664a9457ea5f1c367d2cd746c P e32364b73e254343b4bb6991207c2f85 [term 1 LEADER]: Becoming Leader. State: Replica: e32364b73e254343b4bb6991207c2f85, State: Running, Role: LEADER
I20260812 07:04:38.851292 15919 consensus_queue.cc:237] T a3fe12b664a9457ea5f1c367d2cd746c P e32364b73e254343b4bb6991207c2f85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } }
I20260812 07:04:38.854374 15326 catalog_manager.cc:5495] T a3fe12b664a9457ea5f1c367d2cd746c P e32364b73e254343b4bb6991207c2f85 reported cstate change: term changed from 0 to 1, leader changed from <none> to e32364b73e254343b4bb6991207c2f85 (127.6.61.2). New cstate: current_term: 1 leader_uuid: "e32364b73e254343b4bb6991207c2f85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e32364b73e254343b4bb6991207c2f85" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 35663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 } health_report { overall_health: UNKNOWN } } }
I20260812 07:04:38.892750 16017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:38.887092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33478 (local address 127.6.61.2:35663)
0812 07:04:38.887712 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:38.887717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:38.887731 (+ 14us) server_negotiation.cc:407] Connection header received
0812 07:04:38.887781 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:38.887785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:38.887862 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:38.887988 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:04:38.888439 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:38.889278 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:38.890704 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:38.891285 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:38.891427 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:38.891931 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:04:38.892057 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:04:38.892158 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:38.892315 (+ 157us) server_negotiation.cc:299] Negotiation successful
0812 07:04:38.892401 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":311,"threads_started":1}
I20260812 07:04:38.902829 15827 raft_consensus.cc:1261] T a3fe12b664a9457ea5f1c367d2cd746c P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 1 FOLLOWER]: Refusing update from remote peer e32364b73e254343b4bb6991207c2f85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:04:38.903472 15503 raft_consensus.cc:1261] T a3fe12b664a9457ea5f1c367d2cd746c P 2da9547064764ae9bfb18d37ce320d7c [term 1 FOLLOWER]: Refusing update from remote peer e32364b73e254343b4bb6991207c2f85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:04:38.903463 16014 consensus_queue.cc:1035] T a3fe12b664a9457ea5f1c367d2cd746c P e32364b73e254343b4bb6991207c2f85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59b3abeeb4124526b938e9dc4cf9a7c0" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:04:38.904045 15919 consensus_queue.cc:1035] T a3fe12b664a9457ea5f1c367d2cd746c P e32364b73e254343b4bb6991207c2f85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2da9547064764ae9bfb18d37ce320d7c" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:04:38.974247 15323 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:04:38.974545 15323 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:04:38.978032 15323 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc0633cfe3c74aa6a8a6d4fbe3cec982: Sending DeleteTablet for 3 replicas of tablet a3fe12b664a9457ea5f1c367d2cd746c
I20260812 07:04:38.978976 15481 tablet_service.cc:1514] Processing DeleteTablet for tablet a3fe12b664a9457ea5f1c367d2cd746c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:04:38 UTC) from {username='oryx', principal='oryx/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:60892
I20260812 07:04:38.978986 15641 tablet_service.cc:1514] Processing DeleteTablet for tablet a3fe12b664a9457ea5f1c367d2cd746c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:04:38 UTC) from {username='oryx', principal='oryx/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:33444
I20260812 07:04:38.979388 15983 tablet_replica.cc:330] T a3fe12b664a9457ea5f1c367d2cd746c P 2da9547064764ae9bfb18d37ce320d7c: stopping tablet replica
I20260812 07:04:38.979604 15983 raft_consensus.cc:2229] T a3fe12b664a9457ea5f1c367d2cd746c P 2da9547064764ae9bfb18d37ce320d7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:04:38.979741 15795 tablet_service.cc:1514] Processing DeleteTablet for tablet a3fe12b664a9457ea5f1c367d2cd746c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:04:38 UTC) from {username='oryx', principal='oryx/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:51874
I20260812 07:04:38.979844 15983 raft_consensus.cc:2258] T a3fe12b664a9457ea5f1c367d2cd746c P 2da9547064764ae9bfb18d37ce320d7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:04:38.980100 15985 tablet_replica.cc:330] T a3fe12b664a9457ea5f1c367d2cd746c P 59b3abeeb4124526b938e9dc4cf9a7c0: stopping tablet replica
I20260812 07:04:38.980108 15982 tablet_replica.cc:330] T a3fe12b664a9457ea5f1c367d2cd746c P e32364b73e254343b4bb6991207c2f85: stopping tablet replica
I20260812 07:04:38.980293 15985 raft_consensus.cc:2229] T a3fe12b664a9457ea5f1c367d2cd746c P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:04:38.980342 15982 raft_consensus.cc:2229] T a3fe12b664a9457ea5f1c367d2cd746c P e32364b73e254343b4bb6991207c2f85 [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:04:38.980546 15985 raft_consensus.cc:2258] T a3fe12b664a9457ea5f1c367d2cd746c P 59b3abeeb4124526b938e9dc4cf9a7c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:04:38.980788 15982 raft_consensus.cc:2258] T a3fe12b664a9457ea5f1c367d2cd746c P e32364b73e254343b4bb6991207c2f85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:04:38.981289 15983 ts_tablet_manager.cc:1905] T a3fe12b664a9457ea5f1c367d2cd746c P 2da9547064764ae9bfb18d37ce320d7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:04:38.982227 15982 ts_tablet_manager.cc:1905] T a3fe12b664a9457ea5f1c367d2cd746c P e32364b73e254343b4bb6991207c2f85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:04:38.982385 15985 ts_tablet_manager.cc:1905] T a3fe12b664a9457ea5f1c367d2cd746c P 59b3abeeb4124526b938e9dc4cf9a7c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:04:38.988515 15983 ts_tablet_manager.cc:1918] T a3fe12b664a9457ea5f1c367d2cd746c P 2da9547064764ae9bfb18d37ce320d7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 07:04:38.988662 15983 log.cc:1196] T a3fe12b664a9457ea5f1c367d2cd746c P 2da9547064764ae9bfb18d37ce320d7c: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-0/wal/wals/a3fe12b664a9457ea5f1c367d2cd746c
I20260812 07:04:38.989192 15983 ts_tablet_manager.cc:1939] T a3fe12b664a9457ea5f1c367d2cd746c P 2da9547064764ae9bfb18d37ce320d7c: Deleting consensus metadata
I20260812 07:04:38.989282 15982 ts_tablet_manager.cc:1918] T a3fe12b664a9457ea5f1c367d2cd746c P e32364b73e254343b4bb6991207c2f85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 07:04:38.989415 15982 log.cc:1196] T a3fe12b664a9457ea5f1c367d2cd746c P e32364b73e254343b4bb6991207c2f85: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-1/wal/wals/a3fe12b664a9457ea5f1c367d2cd746c
I20260812 07:04:38.989409 15985 ts_tablet_manager.cc:1918] T a3fe12b664a9457ea5f1c367d2cd746c P 59b3abeeb4124526b938e9dc4cf9a7c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 07:04:38.989542 15985 log.cc:1196] T a3fe12b664a9457ea5f1c367d2cd746c P 59b3abeeb4124526b938e9dc4cf9a7c0: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518208686283-6388-0/minicluster-data/ts-2/wal/wals/a3fe12b664a9457ea5f1c367d2cd746c
I20260812 07:04:38.989876 15982 ts_tablet_manager.cc:1939] T a3fe12b664a9457ea5f1c367d2cd746c P e32364b73e254343b4bb6991207c2f85: Deleting consensus metadata
I20260812 07:04:38.990028 15985 ts_tablet_manager.cc:1939] T a3fe12b664a9457ea5f1c367d2cd746c P 59b3abeeb4124526b938e9dc4cf9a7c0: Deleting consensus metadata
I20260812 07:04:38.991146 15309 catalog_manager.cc:4841] TS e32364b73e254343b4bb6991207c2f85 (127.6.61.2:35663): tablet a3fe12b664a9457ea5f1c367d2cd746c (table test-table [id=9fb27b86b0004e228530c9e809049174]) successfully deleted
I20260812 07:04:38.991495 15307 catalog_manager.cc:4841] TS 59b3abeeb4124526b938e9dc4cf9a7c0 (127.6.61.3:33203): tablet a3fe12b664a9457ea5f1c367d2cd746c (table test-table [id=9fb27b86b0004e228530c9e809049174]) successfully deleted
I20260812 07:04:38.992388 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 15406
I20260812 07:04:38.992372 15305 catalog_manager.cc:4841] TS 2da9547064764ae9bfb18d37ce320d7c (127.6.61.1:34611): tablet a3fe12b664a9457ea5f1c367d2cd746c (table test-table [id=9fb27b86b0004e228530c9e809049174]) successfully deleted
Aug 12 07:04:39 dist-test-slave-69rg krb5kdc[15268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.61.1@KRBTEST.COM
I20260812 07:04:39.024860 16025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:38.993443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60904 (local address 127.6.61.1:34611)
0812 07:04:39.007099 (+ 13656us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:39.007105 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:39.007123 (+ 18us) server_negotiation.cc:407] Connection header received
0812 07:04:39.007178 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:39.007181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:39.007244 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:39.007379 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:39.008929 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:39.010043 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:39.011424 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:39.011932 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:39.012077 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:39.016972 (+ 4895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:39.016990 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:39.017010 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:39.017055 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:39.020945 (+ 3890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:39.021848 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:39.021859 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:39.021866 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:39.022029 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:39.022426 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:39.022430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:39.022433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:39.022881 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:39.023143 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:39.024268 (+ 1125us) server_negotiation.cc:299] Negotiation successful
0812 07:04:39.024496 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13472,"thread_start_us":13324,"threads_started":1}
I20260812 07:04:39.026930 15554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:39.163107 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 15406
I20260812 07:04:39.191226 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 15565
Aug 12 07:04:39 dist-test-slave-69rg krb5kdc[15268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.61.2@KRBTEST.COM
I20260812 07:04:39.208200 16017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:39.192117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40690 (local address 127.6.61.2:35663)
0812 07:04:39.192320 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:39.192326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:39.192367 (+ 41us) server_negotiation.cc:407] Connection header received
0812 07:04:39.192566 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:39.192572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:39.192666 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:39.192776 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:39.194206 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:39.195272 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:39.196497 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:39.197127 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:39.197254 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:39.201767 (+ 4513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:39.201788 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:39.201842 (+ 54us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:39.201887 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:39.205235 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:39.206098 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:39.206106 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:39.206109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:39.206253 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:39.206638 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:39.206642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:39.206645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:39.206989 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:39.207253 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:39.207597 (+ 344us) server_negotiation.cc:299] Negotiation successful
0812 07:04:39.207780 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 07:04:39.208945 15711 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:39.364446 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 15565
I20260812 07:04:39.396508 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 15720
Aug 12 07:04:39 dist-test-slave-69rg krb5kdc[15268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.61.3@KRBTEST.COM
I20260812 07:04:39.416486 16030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:39.397417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52220 (local address 127.6.61.3:33203)
0812 07:04:39.397963 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:39.397969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:39.397992 (+ 23us) server_negotiation.cc:407] Connection header received
0812 07:04:39.398050 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:39.398054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:39.398145 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:39.398268 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:39.399616 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:39.400367 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:39.401618 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:39.402076 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:39.402218 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:39.410693 (+ 8475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:39.410710 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:39.410728 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:39.410765 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:39.413499 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:39.414337 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:39.414344 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:39.414346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:39.414477 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:39.414855 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:39.414860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:39.414863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:39.415245 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:39.415561 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:39.415893 (+ 332us) server_negotiation.cc:299] Negotiation successful
0812 07:04:39.416146 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":266,"threads_started":1}
I20260812 07:04:39.417299 15876 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:39.542955 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 15720
I20260812 07:04:39.572109 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 15284
I20260812 07:04:39.573454 15370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:39.711067 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 15284
2026-08-12T07:04:39Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6378 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:04:39 dist-test-slave-69rg krb5kdc[16036](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:04:39 dist-test-slave-69rg krb5kdc[16036](info): set up 2 sockets
Aug 12 07:04:39 dist-test-slave-69rg krb5kdc[16036](info): commencing operation
krb5kdc: starting...
W20260812 07:04:41.019269 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.258s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:04:41 dist-test-slave-69rg krb5kdc[16036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:04:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:04:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
I20260812 07:04:41.309221 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:45583
--webserver_interface=127.6.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:45583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:41.652521 16050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:41.652859 16050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:41.653025 16050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:41.660360 16050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:04:41.660521 16050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:41.660629 16050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:41.660732 16050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:04:41.660815 16050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:04:41.671412 16050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:45583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:45583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:41.672835 16050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:41.674247 16050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:41.686401 16058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:41.686448 16055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:41.686359 16056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:41.687201 16050 server_base.cc:1034] running on GCE node
I20260812 07:04:41.688035 16050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:41.689419 16050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:41.690665 16050 hybrid_clock.cc:648] HybridClock initialized: now 1786518281690605 us; error 48 us; skew 500 ppm
I20260812 07:04:41.691020 16050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-12T07:04:41Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2056 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:04:41 dist-test-slave-69rg krb5kdc[16066](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:04:41 dist-test-slave-69rg krb5kdc[16066](info): set up 2 sockets
Aug 12 07:04:41 dist-test-slave-69rg krb5kdc[16066](info): commencing operation
krb5kdc: starting...
W20260812 07:04:42.979782 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.162s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:04:43 dist-test-slave-69rg krb5kdc[16066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:04:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:04:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
I20260812 07:04:43.255270 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:34289
--webserver_interface=127.6.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:43041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:34289
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:43.594341 16083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:43.594637 16083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:43.594800 16083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:43.602025 16083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:04:43.602134 16083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:43.602265 16083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:43.602370 16083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:04:43.602435 16083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:04:43.612243 16083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:43041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:34289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:34289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:43.614073 16083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:43.615384 16083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:43.627406 16089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:43.627454 16090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:43.627391 16092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:43.628544 16083 server_base.cc:1034] running on GCE node
I20260812 07:04:43.629300 16083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:43.630712 16083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:43.631939 16083 hybrid_clock.cc:648] HybridClock initialized: now 1786518283631870 us; error 50 us; skew 500 ppm
I20260812 07:04:43.632354 16083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:04:43 dist-test-slave-69rg krb5kdc[16066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518283, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:43.635646 16083 init.cc:377] Logged in from keytab as kudu/127.6.61.62@KRBTEST.COM (short username kudu)
I20260812 07:04:43.637063 16083 webserver.cc:460] Webserver started at http://127.6.61.62:46809/ using document root <none> and password file <none>
I20260812 07:04:43.637753 16083 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:43.637936 16083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:43.638289 16083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:43.641458 16083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "e4cd0c171aaa40679f7d1e17be43a89e"
format_stamp: "Formatted at 2026-08-12 07:04:43 on dist-test-slave-69rg"
server_key: "f66405f1a43e2bbd5e6e1ad458e1ff26"
server_key_iv: "df50328513673bcd4cbf2a22ae1b04d0"
server_key_version: "encryptionkey@0"
I20260812 07:04:43.642267 16083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "e4cd0c171aaa40679f7d1e17be43a89e"
format_stamp: "Formatted at 2026-08-12 07:04:43 on dist-test-slave-69rg"
server_key: "f66405f1a43e2bbd5e6e1ad458e1ff26"
server_key_iv: "df50328513673bcd4cbf2a22ae1b04d0"
server_key_version: "encryptionkey@0"
I20260812 07:04:43.647239 16083 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260812 07:04:43.650539 16099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:43.651822 16083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:04:43.652002 16083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "e4cd0c171aaa40679f7d1e17be43a89e"
format_stamp: "Formatted at 2026-08-12 07:04:43 on dist-test-slave-69rg"
server_key: "f66405f1a43e2bbd5e6e1ad458e1ff26"
server_key_iv: "df50328513673bcd4cbf2a22ae1b04d0"
server_key_version: "encryptionkey@0"
I20260812 07:04:43.652190 16083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:43.688341 16083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:43.692432 16083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:43.692723 16083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:43.711783 16083 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:34289
I20260812 07:04:43.711843 16159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:34289 every 8 connection(s)
I20260812 07:04:43.713243 16083 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:04:43.717093 16160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:43.722915 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 16083
I20260812 07:04:43.723145 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:04:43.723783 6388 external_mini_cluster.cc:1428] Setting key dc4e2fdb8e140197744430fe72cbd50c
I20260812 07:04:43.729351 16160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e: Bootstrap starting.
I20260812 07:04:43.733199 16160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:43.734472 16160 log.cc:824] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:43.737771 16160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e: No bootstrap required, opened a new log
Aug 12 07:04:43 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:04:43.748334 16160 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cd0c171aaa40679f7d1e17be43a89e" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34289 } }
I20260812 07:04:43.748805 16160 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:43.748945 16160 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cd0c171aaa40679f7d1e17be43a89e, State: Initialized, Role: FOLLOWER
I20260812 07:04:43.749686 16160 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cd0c171aaa40679f7d1e17be43a89e" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34289 } }
I20260812 07:04:43.749984 16164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:43.726026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51380 (local address 127.6.61.62:34289)
0812 07:04:43.726935 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:43.726952 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:43.727005 (+ 53us) server_negotiation.cc:407] Connection header received
0812 07:04:43.729460 (+ 2455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:43.729494 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:43.730056 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:43.730497 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:43.731881 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:43.733936 (+ 2055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:43.735480 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:43.736447 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:43.736638 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:43.740944 (+ 4306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:43.740955 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:43.740980 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:43.741034 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:43.745201 (+ 4167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:43.745957 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:43.745966 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:43.745976 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:43.746195 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:43.746624 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:43.746629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:43.746632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:43.747373 (+ 741us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:43.747668 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:43.748005 (+ 337us) server_negotiation.cc:299] Negotiation successful
0812 07:04:43.748475 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":359,"threads_started":1}
I20260812 07:04:43.750022 16160 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:04:43.750152 16160 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:04:43.750360 16160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:43.752774 16160 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cd0c171aaa40679f7d1e17be43a89e" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34289 } }
I20260812 07:04:43.753264 16160 leader_election.cc:302] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4cd0c171aaa40679f7d1e17be43a89e; no voters:
I20260812 07:04:43.753914 16160 leader_election.cc:288] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:04:43.754041 16166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:04:43.754765 16166 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e [term 1 LEADER]: Becoming Leader. State: Replica: e4cd0c171aaa40679f7d1e17be43a89e, State: Running, Role: LEADER
I20260812 07:04:43.755323 16166 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cd0c171aaa40679f7d1e17be43a89e" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34289 } }
I20260812 07:04:43.756189 16160 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:04:43.758298 16167 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4cd0c171aaa40679f7d1e17be43a89e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cd0c171aaa40679f7d1e17be43a89e" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34289 } } }
I20260812 07:04:43.758708 16167 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:43.758270 16169 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4cd0c171aaa40679f7d1e17be43a89e. Latest consensus state: current_term: 1 leader_uuid: "e4cd0c171aaa40679f7d1e17be43a89e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cd0c171aaa40679f7d1e17be43a89e" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34289 } } }
I20260812 07:04:43.763091 16169 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:43.763720 16172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:04:43.768925 16172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:04:43.778031 16172 catalog_manager.cc:1347] Generated new cluster ID: 4ac17767e2d94ff496dcc40c75d96a47
I20260812 07:04:43.778154 16172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:04:43.788887 16172 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:04:43.790822 16172 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:04:43.801355 16172 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e: Generated new TSK 0
I20260812 07:04:43.802421 16172 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
I20260812 07:04:43.874353 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.1:0
--local_ip_for_outbound_sockets=127.6.61.1
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:34289
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:43041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:44.234633 16190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:44.235088 16190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:44.235299 16190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:44.246201 16190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:44.246330 16190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:44.246460 16190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.1
I20260812 07:04:44.257272 16190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:43041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:34289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.1
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:44.258710 16190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:44.259989 16190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:44.273662 16196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:44.273634 16199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:44.273590 16197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:44.274507 16190 server_base.cc:1034] running on GCE node
I20260812 07:04:44.275187 16190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:44.276126 16190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:44.277387 16190 hybrid_clock.cc:648] HybridClock initialized: now 1786518284277329 us; error 50 us; skew 500 ppm
I20260812 07:04:44.277731 16190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:04:44 dist-test-slave-69rg krb5kdc[16066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518284, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:44.280987 16190 init.cc:377] Logged in from keytab as kudu/127.6.61.1@KRBTEST.COM (short username kudu)
I20260812 07:04:44.282505 16190 webserver.cc:460] Webserver started at http://127.6.61.1:40171/ using document root <none> and password file <none>
I20260812 07:04:44.283159 16190 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:44.283285 16190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:44.283604 16190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:44.286664 16190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-0/data/instance:
uuid: "bfd99b6b43664bbdb0cc4ebe8e6a2684"
format_stamp: "Formatted at 2026-08-12 07:04:44 on dist-test-slave-69rg"
server_key: "6ce42ca70575229282fa3317eef98016"
server_key_iv: "9dcb3da93e538473c581ad1a3d2bff6b"
server_key_version: "encryptionkey@0"
I20260812 07:04:44.287447 16190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance:
uuid: "bfd99b6b43664bbdb0cc4ebe8e6a2684"
format_stamp: "Formatted at 2026-08-12 07:04:44 on dist-test-slave-69rg"
server_key: "6ce42ca70575229282fa3317eef98016"
server_key_iv: "9dcb3da93e538473c581ad1a3d2bff6b"
server_key_version: "encryptionkey@0"
I20260812 07:04:44.292902 16190 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 07:04:44.296557 16206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:44.297967 16190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:04:44.298156 16190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-0/wal
uuid: "bfd99b6b43664bbdb0cc4ebe8e6a2684"
format_stamp: "Formatted at 2026-08-12 07:04:44 on dist-test-slave-69rg"
server_key: "6ce42ca70575229282fa3317eef98016"
server_key_iv: "9dcb3da93e538473c581ad1a3d2bff6b"
server_key_version: "encryptionkey@0"
I20260812 07:04:44.298360 16190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:44.329715 16190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:44.334088 16190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:44.334427 16190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:44.335494 16190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:44.336854 16190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:44.336930 16190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:44.337016 16190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:44.337069 16190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:44.371886 16190 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.1:38819
I20260812 07:04:44.371966 16342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.1:38819 every 8 connection(s)
I20260812 07:04:44.373245 16190 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
I20260812 07:04:44.375514 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 16190
I20260812 07:04:44.375746 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance
I20260812 07:04:44.376591 6388 external_mini_cluster.cc:1428] Setting key 46ce068d2f5f08b8a8d0193dc4d3aa3c
Aug 12 07:04:44 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518284, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:04:44.395117 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:44.375998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:42167 (local address 127.6.61.62:34289)
0812 07:04:44.376501 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:44.376508 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:44.377657 (+ 1149us) server_negotiation.cc:407] Connection header received
0812 07:04:44.379644 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:44.379652 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:44.379744 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:44.379866 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:44.381397 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:44.382553 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:44.384233 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:44.384958 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:44.385105 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:44.389205 (+ 4100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:44.389210 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:44.389214 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:44.389248 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:44.391902 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:44.392658 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:44.392665 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:44.392668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:44.392793 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:44.393237 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:44.393242 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:44.393246 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:44.393513 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:44.393672 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:44.394480 (+ 808us) server_negotiation.cc:299] Negotiation successful
0812 07:04:44.394716 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":234,"threads_started":1}
I20260812 07:04:44.396126 16346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:44.376608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.1:42167)
0812 07:04:44.377494 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:44.377539 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:44.379372 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:44.380078 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:44.380087 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:44.380636 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:44.381216 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:44.381233 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:44.382758 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:44.382762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:44.384063 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:44.384076 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:44.385237 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:44.385243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:44.385456 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:44.386270 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:44.386295 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:44.388943 (+ 2648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:44.392062 (+ 3119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:44.392072 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:44.392089 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:44.392534 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:44.392914 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:44.392919 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:44.392923 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:44.393124 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:44.393648 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:44.393656 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:44.394086 (+ 430us) client_negotiation.cc:769] Sending connection context
0812 07:04:44.394368 (+ 282us) client_negotiation.cc:240] Negotiation successful
0812 07:04:44.394927 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":370,"threads_started":1}
I20260812 07:04:44.397899 16343 heartbeater.cc:347] Connected to a master server at 127.6.61.62:34289
I20260812 07:04:44.398239 16343 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:44.398981 16343 heartbeater.cc:511] Master 127.6.61.62:34289 requested a full tablet report, sending...
I20260812 07:04:44.401615 16118 ts_manager.cc:194] Registered new tserver with Master: bfd99b6b43664bbdb0cc4ebe8e6a2684 (127.6.61.1:38819)
I20260812 07:04:44.403292 16118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.1@KRBTEST.COM'} at 127.6.61.1:42167
WARNING: no policy specified for kudu/127.6.61.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
I20260812 07:04:44.438794 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.2:0
--local_ip_for_outbound_sockets=127.6.61.2
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:34289
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:43041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:44.775353 16349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:44.775671 16349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:44.775840 16349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:44.783089 16349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:44.783249 16349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:44.783422 16349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.2
I20260812 07:04:44.793103 16349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:43041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:34289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.2
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:44.794440 16349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:44.795787 16349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:44.809280 16354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:44.809249 16355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:44.809584 16357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:44.810150 16349 server_base.cc:1034] running on GCE node
I20260812 07:04:44.810746 16349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:44.811643 16349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:44.812880 16349 hybrid_clock.cc:648] HybridClock initialized: now 1786518284812823 us; error 45 us; skew 500 ppm
I20260812 07:04:44.813198 16349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:04:44 dist-test-slave-69rg krb5kdc[16066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518284, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:44.816452 16349 init.cc:377] Logged in from keytab as kudu/127.6.61.2@KRBTEST.COM (short username kudu)
I20260812 07:04:44.818015 16349 webserver.cc:460] Webserver started at http://127.6.61.2:44949/ using document root <none> and password file <none>
I20260812 07:04:44.818727 16349 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:44.818874 16349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:44.819200 16349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:44.822201 16349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-1/data/instance:
uuid: "67aa48ed72f44a6392f83cfbddab8ed2"
format_stamp: "Formatted at 2026-08-12 07:04:44 on dist-test-slave-69rg"
server_key: "a38c2a29236e7f39153db46a39d437b6"
server_key_iv: "68d25a58dc3eede598c0cbba1e4a3997"
server_key_version: "encryptionkey@0"
I20260812 07:04:44.823017 16349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance:
uuid: "67aa48ed72f44a6392f83cfbddab8ed2"
format_stamp: "Formatted at 2026-08-12 07:04:44 on dist-test-slave-69rg"
server_key: "a38c2a29236e7f39153db46a39d437b6"
server_key_iv: "68d25a58dc3eede598c0cbba1e4a3997"
server_key_version: "encryptionkey@0"
I20260812 07:04:44.828264 16349 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260812 07:04:44.831888 16366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:44.833185 16349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 07:04:44.833355 16349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-1/wal
uuid: "67aa48ed72f44a6392f83cfbddab8ed2"
format_stamp: "Formatted at 2026-08-12 07:04:44 on dist-test-slave-69rg"
server_key: "a38c2a29236e7f39153db46a39d437b6"
server_key_iv: "68d25a58dc3eede598c0cbba1e4a3997"
server_key_version: "encryptionkey@0"
I20260812 07:04:44.833557 16349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:44.872354 16349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:44.876546 16349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:44.876848 16349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:44.877866 16349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:44.879247 16349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:44.879361 16349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:44.879505 16349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:44.879575 16349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:44.913084 16349 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.2:35135
I20260812 07:04:44.913133 16497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.2:35135 every 8 connection(s)
I20260812 07:04:44.914481 16349 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
I20260812 07:04:44.916102 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 16349
I20260812 07:04:44.916296 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance
I20260812 07:04:44.916911 6388 external_mini_cluster.cc:1428] Setting key 89a60003094455133f179e4013fe1d9c
Aug 12 07:04:44 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518284, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:04:44.938762 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:44.918097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:45147 (local address 127.6.61.62:34289)
0812 07:04:44.918596 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:44.918603 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:44.919701 (+ 1098us) server_negotiation.cc:407] Connection header received
0812 07:04:44.922645 (+ 2944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:44.922649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:44.922720 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:44.922814 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:44.924997 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:44.925741 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:44.928302 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:44.928757 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:44.928874 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:44.932834 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:44.932839 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:44.932843 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:44.932876 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:44.935480 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:44.936234 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:44.936238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:44.936240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:44.936386 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:44.936890 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:44.936896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:44.936898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:44.937190 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:44.937367 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:44.938116 (+ 749us) server_negotiation.cc:299] Negotiation successful
0812 07:04:44.938358 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":254,"threads_started":1}
I20260812 07:04:44.939778 16501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:44.918717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.2:45147)
0812 07:04:44.919520 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:44.919582 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:44.922353 (+ 2771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:44.923088 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:44.923102 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:44.923953 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:44.924787 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:44.924835 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:44.925922 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:44.925930 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:44.928130 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:44.928151 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:44.929002 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:44.929013 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:44.929336 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:44.930180 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:44.930214 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:44.932592 (+ 2378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:44.935630 (+ 3038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:44.935639 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:44.935655 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:44.936094 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:44.936505 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:44.936510 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:44.936513 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:44.936726 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:44.937322 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:44.937331 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:44.937673 (+ 342us) client_negotiation.cc:769] Sending connection context
0812 07:04:44.938045 (+ 372us) client_negotiation.cc:240] Negotiation successful
0812 07:04:44.938619 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":321,"threads_started":1}
I20260812 07:04:44.941349 16498 heartbeater.cc:347] Connected to a master server at 127.6.61.62:34289
I20260812 07:04:44.941665 16498 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:44.942473 16498 heartbeater.cc:511] Master 127.6.61.62:34289 requested a full tablet report, sending...
I20260812 07:04:44.944380 16118 ts_manager.cc:194] Registered new tserver with Master: 67aa48ed72f44a6392f83cfbddab8ed2 (127.6.61.2:35135)
I20260812 07:04:44.945626 16118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.2@KRBTEST.COM'} at 127.6.61.2:45147
WARNING: no policy specified for kudu/127.6.61.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
I20260812 07:04:44.980955 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.3:0
--local_ip_for_outbound_sockets=127.6.61.3
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:34289
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:43041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:45.326023 16504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:45.326333 16504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:45.326432 16504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:45.334013 16504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:45.334136 16504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:45.334267 16504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.3
I20260812 07:04:45.344964 16504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:43041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:34289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.3
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:45.346361 16504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:45.347680 16504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:45.361994 16515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:45.362004 16512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:45.362007 16513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:45.363039 16504 server_base.cc:1034] running on GCE node
I20260812 07:04:45.363700 16504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:45.364722 16504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:45.365970 16504 hybrid_clock.cc:648] HybridClock initialized: now 1786518285365885 us; error 60 us; skew 500 ppm
I20260812 07:04:45.366371 16504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:04:45 dist-test-slave-69rg krb5kdc[16066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518285, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:45.370005 16504 init.cc:377] Logged in from keytab as kudu/127.6.61.3@KRBTEST.COM (short username kudu)
I20260812 07:04:45.371799 16504 webserver.cc:460] Webserver started at http://127.6.61.3:36219/ using document root <none> and password file <none>
I20260812 07:04:45.372786 16504 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:45.372969 16504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:45.373332 16504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:45.377590 16504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-2/data/instance:
uuid: "b4ebf91d52f94e53abe1ecbf337632a6"
format_stamp: "Formatted at 2026-08-12 07:04:45 on dist-test-slave-69rg"
server_key: "9153c8cbe351bb0534c556cda8a87e12"
server_key_iv: "728ca55ac0efa62b0dbc6e4b3219b141"
server_key_version: "encryptionkey@0"
I20260812 07:04:45.378719 16504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance:
uuid: "b4ebf91d52f94e53abe1ecbf337632a6"
format_stamp: "Formatted at 2026-08-12 07:04:45 on dist-test-slave-69rg"
server_key: "9153c8cbe351bb0534c556cda8a87e12"
server_key_iv: "728ca55ac0efa62b0dbc6e4b3219b141"
server_key_version: "encryptionkey@0"
I20260812 07:04:45.384526 16504 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260812 07:04:45.388585 16523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:45.390219 16504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 07:04:45.390422 16504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-2/wal
uuid: "b4ebf91d52f94e53abe1ecbf337632a6"
format_stamp: "Formatted at 2026-08-12 07:04:45 on dist-test-slave-69rg"
server_key: "9153c8cbe351bb0534c556cda8a87e12"
server_key_iv: "728ca55ac0efa62b0dbc6e4b3219b141"
server_key_version: "encryptionkey@0"
I20260812 07:04:45.390615 16504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:45.406365 16343 heartbeater.cc:503] Master 127.6.61.62:34289 was elected leader, sending a full tablet report...
I20260812 07:04:45.406618 16504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:45.411281 16504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:45.411580 16504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:45.412562 16504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:45.413971 16504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:45.414067 16504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:45.414211 16504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:45.414292 16504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:45.448274 16504 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.3:45341
I20260812 07:04:45.448288 16649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.3:45341 every 8 connection(s)
I20260812 07:04:45.449626 16504 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
I20260812 07:04:45.458015 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 16504
I20260812 07:04:45.458202 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance
I20260812 07:04:45.458742 6388 external_mini_cluster.cc:1428] Setting key bb79e2e1c97b912f1eef7ce782825438
Aug 12 07:04:45 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518285, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:04:45.470297 16652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:45.452282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:47169 (local address 127.6.61.62:34289)
0812 07:04:45.452767 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:45.452771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:45.453541 (+ 770us) server_negotiation.cc:407] Connection header received
0812 07:04:45.455504 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:45.455510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:45.455597 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:45.455728 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:45.457339 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:45.458524 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:45.460282 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:45.460864 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:45.460980 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:45.464535 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:45.464540 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:45.464543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:45.464576 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:45.467294 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:45.467995 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:45.468001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:45.468004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:45.468131 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:45.468555 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:45.468561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:45.468564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:45.468839 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:45.469001 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:45.469659 (+ 658us) server_negotiation.cc:299] Negotiation successful
0812 07:04:45.469913 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":252,"threads_started":1}
I20260812 07:04:45.471244 16653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:45.452635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.3:47169)
0812 07:04:45.453374 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:45.453420 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:45.455240 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:45.455982 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:45.455990 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:45.456534 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:45.457136 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:45.457153 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:45.458679 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:45.458683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:45.460120 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:45.460133 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:45.461119 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:45.461127 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:45.461417 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:45.462163 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:45.462185 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:45.464339 (+ 2154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:45.467433 (+ 3094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:45.467442 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:45.467457 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:45.467868 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:45.468240 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:45.468245 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:45.468249 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:45.468438 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:45.468975 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:45.468983 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:45.469297 (+ 314us) client_negotiation.cc:769] Sending connection context
0812 07:04:45.469559 (+ 262us) client_negotiation.cc:240] Negotiation successful
0812 07:04:45.470116 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":283,"threads_started":1}
I20260812 07:04:45.472625 16650 heartbeater.cc:347] Connected to a master server at 127.6.61.62:34289
I20260812 07:04:45.472966 16650 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:45.473908 16650 heartbeater.cc:511] Master 127.6.61.62:34289 requested a full tablet report, sending...
I20260812 07:04:45.475359 16118 ts_manager.cc:194] Registered new tserver with Master: b4ebf91d52f94e53abe1ecbf337632a6 (127.6.61.3:45341)
I20260812 07:04:45.476171 16118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.3@KRBTEST.COM'} at 127.6.61.3:47169
I20260812 07:04:45.486518 6388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.1.keytab.
Entry for principal oryx/127.6.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.1.keytab.
WARNING: no policy specified for oryx/127.6.61.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.61.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.2.keytab.
Entry for principal oryx/127.6.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.2.keytab.
WARNING: no policy specified for oryx/127.6.61.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.61.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.3.keytab.
Entry for principal oryx/127.6.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.3.keytab.
I20260812 07:04:45.652436 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 16190
Aug 12 07:04:45 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.1@KRBTEST.COM
I20260812 07:04:45.669773 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:45.653768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51696 (local address 127.6.61.1:38819)
0812 07:04:45.654641 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:45.654649 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:45.654677 (+ 28us) server_negotiation.cc:407] Connection header received
0812 07:04:45.654744 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:45.654751 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:45.655006 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:45.655140 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:45.656580 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:45.657759 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:45.659108 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:45.659671 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:45.659773 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:45.663894 (+ 4121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:45.663906 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:45.663922 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:45.663962 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:45.666904 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:45.667625 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:45.667633 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:45.667635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:45.667814 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:45.668220 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:45.668223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:45.668225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:45.668585 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:45.668845 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:45.669083 (+ 238us) server_negotiation.cc:299] Negotiation successful
0812 07:04:45.669301 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":531,"threads_started":1}
I20260812 07:04:45.671383 16338 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:45.755447 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 16190
I20260812 07:04:45.776415 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 16349
Aug 12 07:04:45 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.2@KRBTEST.COM
I20260812 07:04:45.792722 16667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:45.777425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46108 (local address 127.6.61.2:35135)
0812 07:04:45.777981 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:45.777989 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:45.778015 (+ 26us) server_negotiation.cc:407] Connection header received
0812 07:04:45.778089 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:45.778096 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:45.778314 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:45.778438 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:45.779721 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:45.780771 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:45.782293 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:45.782864 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:45.782969 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:45.787177 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:45.787189 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:45.787205 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:45.787248 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:45.790298 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:45.790949 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:45.790956 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:45.790958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:45.791085 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:45.791403 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:45.791407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:45.791408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:45.791753 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:45.791980 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:45.792197 (+ 217us) server_negotiation.cc:299] Negotiation successful
0812 07:04:45.792406 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":234,"threads_started":1}
I20260812 07:04:45.794171 16493 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:45.879983 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 16349
I20260812 07:04:45.899598 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 16504
Aug 12 07:04:45 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.3@KRBTEST.COM
I20260812 07:04:45.916467 16670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:45.900666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56214 (local address 127.6.61.3:45341)
0812 07:04:45.901170 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:45.901178 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:45.901208 (+ 30us) server_negotiation.cc:407] Connection header received
0812 07:04:45.901275 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:45.901282 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:45.901550 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:45.901733 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:45.903151 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:45.904197 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:45.905546 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:45.906260 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:45.906409 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:45.910836 (+ 4427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:45.910847 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:45.910861 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:45.910892 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:45.913537 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:45.914341 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:45.914349 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:45.914352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:45.914471 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:45.914858 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:45.914867 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:45.914869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:45.915222 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:45.915545 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:45.915840 (+ 295us) server_negotiation.cc:299] Negotiation successful
0812 07:04:45.916114 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":224,"threads_started":1}
I20260812 07:04:45.917948 16643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:46.001731 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 16504
I20260812 07:04:46.023710 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 16083
I20260812 07:04:46.025048 16153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:04:46.132022 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 16083
I20260812 07:04:46.157653 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:34289
--webserver_interface=127.6.61.62
--webserver_port=46809
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:43041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:34289
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:46.493340 16673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:46.493721 16673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:46.493921 16673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:46.501143 16673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:04:46.501284 16673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:46.501394 16673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:46.501487 16673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:04:46.501569 16673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:04:46.512012 16673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:43041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:34289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:34289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=46809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:46.513408 16673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:46.514719 16673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:46.526921 16680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:46.526911 16681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:46.527017 16683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:46.527684 16673 server_base.cc:1034] running on GCE node
I20260812 07:04:46.528525 16673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:46.529960 16673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:46.531204 16673 hybrid_clock.cc:648] HybridClock initialized: now 1786518286531135 us; error 59 us; skew 500 ppm
I20260812 07:04:46.531584 16673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:04:46 dist-test-slave-69rg krb5kdc[16066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518286, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:46.534874 16673 init.cc:377] Logged in from keytab as kudu/127.6.61.62@KRBTEST.COM (short username kudu)
I20260812 07:04:46.536427 16673 webserver.cc:460] Webserver started at http://127.6.61.62:46809/ using document root <none> and password file <none>
I20260812 07:04:46.537096 16673 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:46.537263 16673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:46.542716 16673 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:04:46.545445 16691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:46.546940 16673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 07:04:46.547132 16673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "e4cd0c171aaa40679f7d1e17be43a89e"
format_stamp: "Formatted at 2026-08-12 07:04:43 on dist-test-slave-69rg"
server_key: "f66405f1a43e2bbd5e6e1ad458e1ff26"
server_key_iv: "df50328513673bcd4cbf2a22ae1b04d0"
server_key_version: "encryptionkey@0"
I20260812 07:04:46.548111 16673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:46.563444 16673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:46.567672 16673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:46.567965 16673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:46.587046 16673 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:34289
I20260812 07:04:46.587085 16753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:34289 every 8 connection(s)
I20260812 07:04:46.588627 16673 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:04:46.593459 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 16673
I20260812 07:04:46.594522 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.1:38819
--local_ip_for_outbound_sockets=127.6.61.1
--tserver_master_addrs=127.6.61.62:34289
--webserver_port=40171
--webserver_interface=127.6.61.1
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:43041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.61.1
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:04:46.596805 16754 sys_catalog.cc:263] Verifying existing consensus state
I20260812 07:04:46.602392 16754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e: Bootstrap starting.
I20260812 07:04:46.623167 16754 log.cc:824] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:46.632468 16754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 07:04:46.633033 16754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e: Bootstrap complete.
I20260812 07:04:46.642452 16754 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cd0c171aaa40679f7d1e17be43a89e" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34289 } }
I20260812 07:04:46.643287 16754 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cd0c171aaa40679f7d1e17be43a89e, State: Initialized, Role: FOLLOWER
I20260812 07:04:46.643899 16754 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cd0c171aaa40679f7d1e17be43a89e" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34289 } }
I20260812 07:04:46.644138 16754 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:04:46.644248 16754 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:04:46.644428 16754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e [term 1 FOLLOWER]: Advancing to term 2
I20260812 07:04:46.648370 16754 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cd0c171aaa40679f7d1e17be43a89e" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34289 } }
I20260812 07:04:46.648869 16754 leader_election.cc:302] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4cd0c171aaa40679f7d1e17be43a89e; no voters:
I20260812 07:04:46.649510 16754 leader_election.cc:288] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 07:04:46.649652 16758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e [term 2 FOLLOWER]: Leader election won for term 2
I20260812 07:04:46.651075 16758 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e [term 2 LEADER]: Becoming Leader. State: Replica: e4cd0c171aaa40679f7d1e17be43a89e, State: Running, Role: LEADER
I20260812 07:04:46.651760 16758 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cd0c171aaa40679f7d1e17be43a89e" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34289 } }
I20260812 07:04:46.651918 16754 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:04:46.654564 16761 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4cd0c171aaa40679f7d1e17be43a89e. Latest consensus state: current_term: 2 leader_uuid: "e4cd0c171aaa40679f7d1e17be43a89e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cd0c171aaa40679f7d1e17be43a89e" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34289 } } }
I20260812 07:04:46.654933 16761 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:46.655972 16759 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e4cd0c171aaa40679f7d1e17be43a89e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cd0c171aaa40679f7d1e17be43a89e" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34289 } } }
I20260812 07:04:46.656347 16759 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:46.666090 16770 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:04:46.672084 16770 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:04:46.673549 16770 catalog_manager.cc:1259] Loaded cluster ID: 4ac17767e2d94ff496dcc40c75d96a47
I20260812 07:04:46.673702 16770 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:04:46.677773 16770 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:04:46.679287 16770 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e4cd0c171aaa40679f7d1e17be43a89e: Loaded TSK: 0
I20260812 07:04:46.680853 16770 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 07:04:46.959956 16756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:46.960268 16756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:46.960438 16756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:46.968905 16756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:46.969027 16756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:46.969221 16756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.1
I20260812 07:04:46.980413 16756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:43041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.1.keytab
--principal=oryx/127.6.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.1:38819
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.61.1
--webserver_port=40171
--tserver_master_addrs=127.6.61.62:34289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.1
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:46.981892 16756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:46.983410 16756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:46.998589 16787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:46.998632 16788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:46.999517 16790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:47.000195 16756 server_base.cc:1034] running on GCE node
I20260812 07:04:47.000816 16756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:47.001965 16756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:47.003201 16756 hybrid_clock.cc:648] HybridClock initialized: now 1786518287003147 us; error 47 us; skew 500 ppm
I20260812 07:04:47.003539 16756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:04:47 dist-test-slave-69rg krb5kdc[16066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518287, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:47.013299 16756 init.cc:377] Logged in from keytab as oryx/127.6.61.1@KRBTEST.COM (short username oryx)
I20260812 07:04:47.015079 16756 webserver.cc:460] Webserver started at http://127.6.61.1:40171/ using document root <none> and password file <none>
I20260812 07:04:47.015807 16756 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:47.015983 16756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:47.021797 16756 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 07:04:47.024591 16799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:47.026001 16756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 07:04:47.026173 16756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-0/wal
uuid: "bfd99b6b43664bbdb0cc4ebe8e6a2684"
format_stamp: "Formatted at 2026-08-12 07:04:44 on dist-test-slave-69rg"
server_key: "6ce42ca70575229282fa3317eef98016"
server_key_iv: "9dcb3da93e538473c581ad1a3d2bff6b"
server_key_version: "encryptionkey@0"
I20260812 07:04:47.027168 16756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:47.040212 16756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:47.044188 16756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:47.044457 16756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:47.045395 16756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:47.046805 16756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:47.046937 16756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:47.047062 16756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:47.047147 16756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:47.081516 16756 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.1:38819
I20260812 07:04:47.081604 16938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.1:38819 every 8 connection(s)
I20260812 07:04:47.083043 16756 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
I20260812 07:04:47.085052 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 16756
I20260812 07:04:47.085899 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.2:35135
--local_ip_for_outbound_sockets=127.6.61.2
--tserver_master_addrs=127.6.61.62:34289
--webserver_port=44949
--webserver_interface=127.6.61.2
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:43041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.61.2
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 07:04:47 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:47 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:47.109551 16943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:47.086505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.1:33825)
0812 07:04:47.091043 (+ 4538us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:47.091094 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:47.093976 (+ 2882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:47.095818 (+ 1842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:47.095830 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:47.096772 (+ 942us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:47.097549 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:47.097568 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:47.100093 (+ 2525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:47.100101 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:47.102088 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:47.102102 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:47.103700 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:47.103705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:47.103916 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:47.104652 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:47.104677 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:47.108591 (+ 3914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":4153,"thread_start_us":3960,"threads_started":1}
W20260812 07:04:47.110245 16942 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:47.086338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:33825 (local address 127.6.61.62:34289)
0812 07:04:47.090557 (+ 4219us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:47.090578 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:47.091264 (+ 686us) server_negotiation.cc:407] Connection header received
0812 07:04:47.094678 (+ 3414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:47.094699 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:47.095189 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:47.095602 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:47.097775 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:47.099883 (+ 2108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:47.102241 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:47.103474 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:47.103686 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:47.109069 (+ 5383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.1:33825: BlockingRecv error: recv got EOF from 127.6.61.1:33825 (error 108)
Metrics: {"server-negotiator.queue_time_us":3922,"thread_start_us":3665,"threads_started":1}
W20260812 07:04:47.110309 16939 heartbeater.cc:644] Failed to heartbeat to 127.6.61.62:34289 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.61.62:34289: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Aug 12 07:04:47 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:47 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:47.121273 16943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:47.111015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.1:34205)
0812 07:04:47.111244 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:47.111260 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:47.111353 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:47.111794 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:47.111802 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:47.112269 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:47.112493 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:47.112508 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:47.113504 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:47.113513 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:47.115287 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:47.115306 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:47.115973 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:47.115982 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:47.116210 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:47.116844 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:47.116875 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:47.120897 (+ 4022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127}
W20260812 07:04:47.121292 16942 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:47.111078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:34205 (local address 127.6.61.62:34289)
0812 07:04:47.111513 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:47.111518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:47.111531 (+ 13us) server_negotiation.cc:407] Connection header received
0812 07:04:47.111576 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:47.111579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:47.111635 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:47.111709 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:47.112641 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:47.113357 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:47.115438 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:47.115816 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:47.115911 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:47.121108 (+ 5197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.1:34205: BlockingRecv error: recv got EOF from 127.6.61.1:34205 (error 108)
Metrics: {"server-negotiator.queue_time_us":344}
Aug 12 07:04:47 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:47 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:47.130885 16943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:47.122224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.1:55033)
0812 07:04:47.122432 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:47.122454 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:47.122562 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:47.122931 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:47.122935 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:47.123241 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:47.123402 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:47.123410 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:47.124833 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:47.124837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:47.125731 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:47.125742 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:47.126740 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:47.126744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:47.126878 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:47.127525 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:47.127544 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:47.130558 (+ 3014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260812 07:04:47.131090 16942 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:47.122397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:55033 (local address 127.6.61.62:34289)
0812 07:04:47.122588 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:47.122597 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:47.122621 (+ 24us) server_negotiation.cc:407] Connection header received
0812 07:04:47.122698 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:47.122704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:47.122791 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:47.122893 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:47.123537 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:47.124648 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:47.125952 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:47.126598 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:47.126738 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:47.130828 (+ 4090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.1:55033: BlockingRecv error: recv got EOF from 127.6.61.1:55033 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260812 07:04:47.131305 16939 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 07:04:47.456233 16944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:47.456578 16944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:47.456758 16944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:47.464715 16944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:47.464879 16944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:47.465051 16944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.2
I20260812 07:04:47.475023 16944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:43041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.2.keytab
--principal=oryx/127.6.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.2:35135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.61.2
--webserver_port=44949
--tserver_master_addrs=127.6.61.62:34289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.2
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:47.476424 16944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:47.477743 16944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:47.491250 16953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:47.491230 16950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:47.491448 16951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:47.492090 16944 server_base.cc:1034] running on GCE node
I20260812 07:04:47.492648 16944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:47.493548 16944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:47.494776 16944 hybrid_clock.cc:648] HybridClock initialized: now 1786518287494711 us; error 80 us; skew 500 ppm
I20260812 07:04:47.495090 16944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:04:47 dist-test-slave-69rg krb5kdc[16066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518287, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:47.498324 16944 init.cc:377] Logged in from keytab as oryx/127.6.61.2@KRBTEST.COM (short username oryx)
I20260812 07:04:47.499810 16944 webserver.cc:460] Webserver started at http://127.6.61.2:44949/ using document root <none> and password file <none>
I20260812 07:04:47.500480 16944 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:47.500636 16944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:47.505923 16944 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260812 07:04:47.508525 16961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:47.509790 16944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 07:04:47.510020 16944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-1/wal
uuid: "67aa48ed72f44a6392f83cfbddab8ed2"
format_stamp: "Formatted at 2026-08-12 07:04:44 on dist-test-slave-69rg"
server_key: "a38c2a29236e7f39153db46a39d437b6"
server_key_iv: "68d25a58dc3eede598c0cbba1e4a3997"
server_key_version: "encryptionkey@0"
I20260812 07:04:47.511013 16944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:47.533660 16944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:47.538511 16944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:47.538796 16944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:47.539785 16944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:47.541177 16944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:47.541293 16944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:47.541415 16944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:47.541501 16944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:47.574570 16944 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.2:35135
I20260812 07:04:47.574635 17092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.2:35135 every 8 connection(s)
I20260812 07:04:47.576184 16944 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
I20260812 07:04:47.582919 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 16944
I20260812 07:04:47.583572 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.3:45341
--local_ip_for_outbound_sockets=127.6.61.3
--tserver_master_addrs=127.6.61.62:34289
--webserver_port=36219
--webserver_interface=127.6.61.3
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:43041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.61.3
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 07:04:47 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:47 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 07:04:47.592520 16942 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:47.579196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:33407 (local address 127.6.61.62:34289)
0812 07:04:47.579385 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:47.579390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:47.580479 (+ 1089us) server_negotiation.cc:407] Connection header received
0812 07:04:47.582480 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:47.582486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:47.582580 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:47.582692 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:47.584323 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:47.585122 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:47.586805 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:47.587244 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:47.587378 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:47.592253 (+ 4875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.2:33407: BlockingRecv error: recv got EOF from 127.6.61.2:33407 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 07:04:47.593523 17096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:47.579566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.2:33407)
0812 07:04:47.580315 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:47.580352 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:47.582205 (+ 1853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:47.582930 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:47.582939 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:47.583497 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:47.584144 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:47.584162 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:47.585293 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:47.585297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:47.586667 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:47.586679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:47.587487 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:47.587496 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:47.587728 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:47.588407 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:47.588435 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:47.592019 (+ 3584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":316,"threads_started":1}
W20260812 07:04:47.594321 17093 heartbeater.cc:644] Failed to heartbeat to 127.6.61.62:34289 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.61.62:34289: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Aug 12 07:04:47 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:47 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:47.606077 17096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:47.595129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.2:56971)
0812 07:04:47.595400 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:47.595419 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:47.595544 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:47.595913 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:47.595921 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:47.596381 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:47.596595 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:47.596609 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:47.598051 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:47.598059 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:47.599686 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:47.599705 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:47.600736 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:47.600744 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:47.600973 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:47.601676 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:47.601705 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:47.605679 (+ 3974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155}
W20260812 07:04:47.606204 16942 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:47.595235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:56971 (local address 127.6.61.62:34289)
0812 07:04:47.595414 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:47.595420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:47.595474 (+ 54us) server_negotiation.cc:407] Connection header received
0812 07:04:47.595679 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:47.595684 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:47.595762 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:47.595868 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:47.596758 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:47.597891 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:47.599863 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:47.600559 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:47.600688 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:47.605992 (+ 5304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.2:56971: BlockingRecv error: recv got EOF from 127.6.61.2:56971 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 12 07:04:47 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:47 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:47.616009 17096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:47.607012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.2:47667)
0812 07:04:47.607220 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:47.607243 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:47.607351 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:47.607686 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:47.607689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:47.607980 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:47.608127 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:47.608134 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:47.609806 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:47.609811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:47.610764 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:47.610776 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:47.611899 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:47.611903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:47.612027 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:47.612656 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:47.612674 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:47.615774 (+ 3100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260812 07:04:47.616355 16942 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:47.607147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:47667 (local address 127.6.61.62:34289)
0812 07:04:47.607338 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:47.607346 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:47.607369 (+ 23us) server_negotiation.cc:407] Connection header received
0812 07:04:47.607443 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:47.607450 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:47.607536 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:47.607658 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:47.608279 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:47.609602 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:47.610921 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:47.611709 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:47.611874 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:47.616068 (+ 4194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.2:47667: BlockingRecv error: recv got EOF from 127.6.61.2:47667 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260812 07:04:47.616407 17093 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 07:04:47.936828 17097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:47.937088 17097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:47.937301 17097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:47.944566 17097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:47.944672 17097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:47.944870 17097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.3
I20260812 07:04:47.954866 17097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:43041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/krb5kdc/oryx_127.6.61.3.keytab
--principal=oryx/127.6.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.3:45341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.61.3
--webserver_port=36219
--tserver_master_addrs=127.6.61.62:34289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.3
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:04:47.956202 17097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:47.957506 17097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:47.971027 17103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:47.971052 17104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:47.970991 17106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:47.971877 17097 server_base.cc:1034] running on GCE node
I20260812 07:04:47.972438 17097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:47.973367 17097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:47.974577 17097 hybrid_clock.cc:648] HybridClock initialized: now 1786518287974534 us; error 31 us; skew 500 ppm
I20260812 07:04:47.974865 17097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:04:47 dist-test-slave-69rg krb5kdc[16066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518287, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:47.978246 17097 init.cc:377] Logged in from keytab as oryx/127.6.61.3@KRBTEST.COM (short username oryx)
I20260812 07:04:47.979897 17097 webserver.cc:460] Webserver started at http://127.6.61.3:36219/ using document root <none> and password file <none>
I20260812 07:04:47.980592 17097 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:47.980755 17097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:47.986430 17097 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 07:04:47.989295 17114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:47.990701 17097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:04:47.990912 17097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-2/wal
uuid: "b4ebf91d52f94e53abe1ecbf337632a6"
format_stamp: "Formatted at 2026-08-12 07:04:45 on dist-test-slave-69rg"
server_key: "9153c8cbe351bb0534c556cda8a87e12"
server_key_iv: "728ca55ac0efa62b0dbc6e4b3219b141"
server_key_version: "encryptionkey@0"
I20260812 07:04:47.991923 17097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:48.013157 17097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:48.018112 17097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:48.018416 17097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:48.019455 17097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:48.020900 17097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:48.021026 17097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:48.021166 17097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:48.021258 17097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:48.055789 17097 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.3:45341
I20260812 07:04:48.055878 17247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.3:45341 every 8 connection(s)
I20260812 07:04:48.057269 17097 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
I20260812 07:04:48.067087 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 17097
Aug 12 07:04:48 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:48 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 07:04:48.080808 16942 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:48.060307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:59305 (local address 127.6.61.62:34289)
0812 07:04:48.060616 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:48.060622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:48.064602 (+ 3980us) server_negotiation.cc:407] Connection header received
0812 07:04:48.067698 (+ 3096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:48.067723 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:48.067842 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:48.067976 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:48.070292 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:48.071479 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:48.073644 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:48.074398 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:48.074554 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:48.080432 (+ 5878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.3:59305: BlockingRecv error: recv got EOF from 127.6.61.3:59305 (error 108)
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":85}
I20260812 07:04:48.081542 17252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:48.060883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.3:59305)
0812 07:04:48.064392 (+ 3509us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:48.064456 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:48.067283 (+ 2827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:48.068422 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:48.068431 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:48.069274 (+ 843us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:48.070104 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:48.070125 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:48.071661 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:48.071665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:48.073493 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:48.073506 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:48.074689 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:48.074699 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:48.075020 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:48.075887 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:48.075917 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:48.080234 (+ 4317us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":3175,"thread_start_us":2984,"threads_started":1}
W20260812 07:04:48.086334 17248 heartbeater.cc:644] Failed to heartbeat to 127.6.61.62:34289 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.61.62:34289: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
I20260812 07:04:48.086759 17254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:48.068595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51384 (local address 127.6.61.62:34289)
0812 07:04:48.069389 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:48.069397 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:48.069418 (+ 21us) server_negotiation.cc:407] Connection header received
0812 07:04:48.069469 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:48.069473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:48.069535 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:48.069640 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:48.071651 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:48.072840 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:48.074574 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:48.075332 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:48.075457 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:48.078711 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:48.078733 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:48.078753 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:48.078793 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:48.082627 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:48.083342 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:48.083352 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:48.083360 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:48.083589 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:48.084007 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:48.084012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:48.084015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:48.084669 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:48.085025 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:48.085435 (+ 410us) server_negotiation.cc:299] Negotiation successful
0812 07:04:48.085842 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":675,"thread_start_us":535,"threads_started":1}
Aug 12 07:04:48 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:48 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:48.095629 17252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:48.087022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.3:38157)
0812 07:04:48.087279 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:48.087300 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:48.087424 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:48.088002 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:48.088010 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:48.088342 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:48.088486 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:48.088494 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:48.089788 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:48.089796 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:48.091071 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:48.091084 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:48.091899 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:48.091904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:48.092030 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:48.092557 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:48.092575 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:48.095381 (+ 2806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146}
W20260812 07:04:48.096222 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:48.087129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:38157 (local address 127.6.61.62:34289)
0812 07:04:48.087660 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:48.087666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:48.087683 (+ 17us) server_negotiation.cc:407] Connection header received
0812 07:04:48.087755 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:48.087760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:48.087834 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:48.087942 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:48.088628 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:48.089616 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:48.091225 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:48.091757 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:48.091872 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:48.095693 (+ 3821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.3:38157: BlockingRecv error: recv got EOF from 127.6.61.3:38157 (error 108)
Metrics: {"server-negotiator.queue_time_us":412}
Aug 12 07:04:48 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:48 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:48.104472 17252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:48.096365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.3:34581)
0812 07:04:48.096529 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:48.096552 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:48.096661 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:48.097079 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:48.097084 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:48.097390 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:48.097534 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:48.097542 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:48.098651 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:48.098654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:48.099579 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:48.099591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:48.100257 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:48.100261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:48.100385 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:48.100911 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:48.100928 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:48.104239 (+ 3311us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 07:04:48.104696 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:48.096472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:34581 (local address 127.6.61.62:34289)
0812 07:04:48.096837 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:48.096841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:48.096854 (+ 13us) server_negotiation.cc:407] Connection header received
0812 07:04:48.096895 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:48.096898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:48.096950 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:48.097052 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:48.097697 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:48.098521 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:48.099723 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:48.100137 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:48.100232 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:48.104450 (+ 4218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.3:34581: BlockingRecv error: recv got EOF from 127.6.61.3:34581 (error 108)
Metrics: {"server-negotiator.queue_time_us":220}
W20260812 07:04:48.104781 17248 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 12 07:04:48 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:48 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:48.141196 17256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:48.131954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.1:58571)
0812 07:04:48.132499 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:48.132515 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:48.132672 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:48.133027 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:48.133032 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:48.133546 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:48.133752 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:48.133762 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:48.134908 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:48.134914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:48.135919 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:48.135931 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:48.136709 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:48.136716 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:48.136893 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:48.137503 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:48.137520 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:48.140952 (+ 3432us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":285,"threads_started":1}
W20260812 07:04:48.141364 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:48.132083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:58571 (local address 127.6.61.62:34289)
0812 07:04:48.132325 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:48.132333 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:48.132600 (+ 267us) server_negotiation.cc:407] Connection header received
0812 07:04:48.132807 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:48.132812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:48.132884 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:48.132990 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:48.133948 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:48.134745 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:48.136068 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:48.136520 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:48.136658 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:48.141162 (+ 4504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.1:58571: BlockingRecv error: recv got EOF from 127.6.61.1:58571 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 12 07:04:48 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:48 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:48.626394 17257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:48.617035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.2:48657)
0812 07:04:48.617591 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:48.617606 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:48.617722 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:48.618114 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:48.618119 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:48.618611 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:48.618820 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:48.618829 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:48.619878 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:48.619884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:48.621372 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:48.621390 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:48.622230 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:48.622237 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:48.622420 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:48.623008 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:48.623028 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:48.626052 (+ 3024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":312,"threads_started":1}
W20260812 07:04:48.626574 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:48.617150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:48657 (local address 127.6.61.62:34289)
0812 07:04:48.617348 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:48.617355 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:48.617669 (+ 314us) server_negotiation.cc:407] Connection header received
0812 07:04:48.617883 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:48.617890 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:48.617966 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:48.618068 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:48.618972 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:48.619727 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:48.621524 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:48.622040 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:48.622196 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:48.626298 (+ 4102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.2:48657: BlockingRecv error: recv got EOF from 127.6.61.2:48657 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 12 07:04:49 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:49 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:49.114909 17258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.105457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.3:34261)
0812 07:04:49.106041 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:49.106059 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:49.106195 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:49.106491 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:49.106494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:49.106990 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:49.107207 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:49.107216 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.108368 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.108373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:49.109365 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:49.109378 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.110136 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.110141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:49.110267 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:49.110950 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:49.110969 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:49.114539 (+ 3570us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":265,"threads_started":1}
W20260812 07:04:49.115032 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:49.105553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:34261 (local address 127.6.61.62:34289)
0812 07:04:49.105790 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:49.105797 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:49.106162 (+ 365us) server_negotiation.cc:407] Connection header received
0812 07:04:49.106283 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:49.106287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:49.106358 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:49.106438 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:49.107372 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.108210 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.109523 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.109969 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.110069 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:49.114697 (+ 4628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.3:34261: BlockingRecv error: recv got EOF from 127.6.61.3:34261 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 12 07:04:49 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:49 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:49.150592 17259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.142211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.1:50271)
0812 07:04:49.142765 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:49.142782 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:49.142904 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:49.143220 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:49.143224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:49.143658 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:49.143816 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:49.143826 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.144863 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.144867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:49.145802 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:49.145876 (+ 74us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.146585 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.146589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:49.146707 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:49.147306 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:49.147348 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:49.150277 (+ 2929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":319,"threads_started":1}
W20260812 07:04:49.150905 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:49.142284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:50271 (local address 127.6.61.62:34289)
0812 07:04:49.142483 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:49.142488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:49.142858 (+ 370us) server_negotiation.cc:407] Connection header received
0812 07:04:49.143005 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:49.143008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:49.143073 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:49.143157 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:49.143969 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.144734 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.146013 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.146444 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.146542 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:49.150586 (+ 4044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.1:50271: BlockingRecv error: recv got EOF from 127.6.61.1:50271 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 12 07:04:49 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:49 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:49.635953 17260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.627466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.2:54041)
0812 07:04:49.628038 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:49.628056 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:49.628173 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:49.628525 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:49.628529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:49.628954 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:49.629104 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:49.629112 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.630235 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.630240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:49.631293 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:49.631307 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.632109 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.632114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:49.632238 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:49.632862 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:49.632880 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:49.635695 (+ 2815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":336,"threads_started":1}
W20260812 07:04:49.636113 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:49.627576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:54041 (local address 127.6.61.62:34289)
0812 07:04:49.627779 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:49.627784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:49.628130 (+ 346us) server_negotiation.cc:407] Connection header received
0812 07:04:49.628309 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:49.628313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:49.628386 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:49.628473 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:49.629246 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.630054 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.631420 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.631898 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.632034 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:49.635899 (+ 3865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.2:54041: BlockingRecv error: recv got EOF from 127.6.61.2:54041 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 12 07:04:50 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:50 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:50.125098 17261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:50.115982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.3:37161)
0812 07:04:50.116568 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:50.116585 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:50.116736 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:50.117158 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:50.117166 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:50.117633 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:50.117791 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:50.117801 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:50.119071 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:50.119077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:50.120020 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:50.120033 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:50.120813 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:50.120820 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:50.120995 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:50.121629 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:50.121649 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:50.124711 (+ 3062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":320,"threads_started":1}
W20260812 07:04:50.125204 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:50.116222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:37161 (local address 127.6.61.62:34289)
0812 07:04:50.116423 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:50.116432 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:50.116677 (+ 245us) server_negotiation.cc:407] Connection header received
0812 07:04:50.116880 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:50.116886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:50.116974 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:50.117109 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:50.118015 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:50.118908 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:50.120189 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:50.120649 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:50.120787 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:50.124949 (+ 4162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.3:37161: BlockingRecv error: recv got EOF from 127.6.61.3:37161 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 12 07:04:50 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:50 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:50.161161 17262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:50.151841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.1:34193)
0812 07:04:50.152446 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:50.152467 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:50.152659 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:50.153096 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:50.153101 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:50.153656 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:50.153929 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:50.153947 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:50.155037 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:50.155042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:50.155996 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:50.156009 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:50.156771 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:50.156776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:50.156915 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:50.157633 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:50.157652 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:50.160790 (+ 3138us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":303,"threads_started":1}
W20260812 07:04:50.161255 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:50.151930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:34193 (local address 127.6.61.62:34289)
0812 07:04:50.152123 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:50.152131 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:50.152571 (+ 440us) server_negotiation.cc:407] Connection header received
0812 07:04:50.152811 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:50.152816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:50.152914 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:50.153020 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:50.154101 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:50.154878 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:50.156184 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:50.156605 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:50.156710 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:50.161010 (+ 4300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.1:34193: BlockingRecv error: recv got EOF from 127.6.61.1:34193 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 12 07:04:50 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:50 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:50.645741 17263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:50.637030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.2:37465)
0812 07:04:50.637591 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:50.637611 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:50.637748 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:50.638167 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:50.638171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:50.638538 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:50.638705 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:50.638714 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:50.639788 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:50.639791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:50.640720 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:50.640731 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:50.641490 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:50.641495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:50.641628 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:50.642226 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:50.642244 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:50.645376 (+ 3132us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":320,"threads_started":1}
W20260812 07:04:50.645948 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:50.637081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:37465 (local address 127.6.61.62:34289)
0812 07:04:50.637279 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:50.637284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:50.637696 (+ 412us) server_negotiation.cc:407] Connection header received
0812 07:04:50.637953 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:50.637957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:50.638025 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:50.638137 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:50.638867 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:50.639652 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:50.640862 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:50.641311 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:50.641456 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:50.645618 (+ 4162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.2:37465: BlockingRecv error: recv got EOF from 127.6.61.2:37465 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 12 07:04:51 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:51 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:51.135509 17264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:51.126269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.3:35597)
0812 07:04:51.126875 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:51.126892 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:51.127023 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:51.127351 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:51.127355 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:51.127707 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:51.127901 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:51.127911 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:51.128949 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:51.128953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:51.130229 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:51.130247 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:51.131279 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:51.131285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:51.131485 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:51.132138 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:51.132157 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:51.135225 (+ 3068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":314,"threads_started":1}
W20260812 07:04:51.135785 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:51.126358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:35597 (local address 127.6.61.62:34289)
0812 07:04:51.126552 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:51.126557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:51.126976 (+ 419us) server_negotiation.cc:407] Connection header received
0812 07:04:51.127138 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:51.127142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:51.127207 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:51.127295 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:51.128042 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:51.128830 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:51.130417 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:51.131123 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:51.131226 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:51.135462 (+ 4236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.3:35597: BlockingRecv error: recv got EOF from 127.6.61.3:35597 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 12 07:04:51 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:51 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:51.171290 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:51.162137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.1:42305)
0812 07:04:51.162631 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:51.162648 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:51.162766 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:51.163120 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:51.163124 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:51.163515 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:51.163692 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:51.163701 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:51.164755 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:51.164759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:51.165876 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:51.165891 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:51.166700 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:51.166706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:51.166851 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:51.167642 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:51.167669 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:51.170939 (+ 3270us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":260,"threads_started":1}
W20260812 07:04:51.171298 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:51.162225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:42305 (local address 127.6.61.62:34289)
0812 07:04:51.162478 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:51.162484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:51.162726 (+ 242us) server_negotiation.cc:407] Connection header received
0812 07:04:51.162910 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:51.162915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:51.162984 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:51.163077 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:51.163833 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:51.164612 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:51.166008 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:51.166472 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:51.166670 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:51.171104 (+ 4434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.1:42305: BlockingRecv error: recv got EOF from 127.6.61.1:42305 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Aug 12 07:04:51 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:51 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:51.655509 17266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:51.646787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.2:34755)
0812 07:04:51.647403 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:51.647420 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:51.647538 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:51.647874 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:51.647877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:51.648248 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:51.648428 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:51.648436 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:51.649480 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:51.649484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:51.650479 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:51.650491 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:51.651165 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:51.651170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:51.651299 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:51.652012 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:51.652031 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:51.655258 (+ 3227us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":290,"threads_started":1}
W20260812 07:04:51.655766 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:51.646872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:34755 (local address 127.6.61.62:34289)
0812 07:04:51.647205 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:51.647212 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:51.647522 (+ 310us) server_negotiation.cc:407] Connection header received
0812 07:04:51.647655 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:51.647659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:51.647726 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:51.647813 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:51.648576 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:51.649363 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:51.650617 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:51.651037 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:51.651149 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:51.655471 (+ 4322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.2:34755: BlockingRecv error: recv got EOF from 127.6.61.2:34755 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 12 07:04:52 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:52 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:52.152261 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:52.136467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.3:56255)
0812 07:04:52.136950 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:52.136965 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:52.137134 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:52.137502 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:52.137506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:52.137970 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:52.138144 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:52.138154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:52.139205 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:52.139211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:52.140182 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:52.140194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:52.140894 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:52.140900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:52.141096 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:52.148750 (+ 7654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:52.148770 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:52.151985 (+ 3215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":255,"threads_started":1}
W20260812 07:04:52.152472 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:52.136496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:56255 (local address 127.6.61.62:34289)
0812 07:04:52.136713 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:52.136722 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:52.137052 (+ 330us) server_negotiation.cc:407] Connection header received
0812 07:04:52.137286 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:52.137290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:52.137360 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:52.137466 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:52.138291 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:52.139067 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:52.140325 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:52.140737 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:52.140881 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:52.152186 (+ 11305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.3:56255: BlockingRecv error: recv got EOF from 127.6.61.3:56255 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 12 07:04:52 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:52 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:52.181077 17268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:52.172382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.1:49525)
0812 07:04:52.173007 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:52.173024 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:52.173249 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:52.173606 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:52.173611 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:52.174176 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:52.174349 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:52.174358 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:52.175458 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:52.175464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:52.176418 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:52.176430 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:52.177213 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:52.177219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:52.177398 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:52.178018 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:52.178036 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:52.180834 (+ 2798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":334,"threads_started":1}
W20260812 07:04:52.181288 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:52.172415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:49525 (local address 127.6.61.62:34289)
0812 07:04:52.172576 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:52.172581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:52.173170 (+ 589us) server_negotiation.cc:407] Connection header received
0812 07:04:52.173377 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:52.173383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:52.173460 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:52.173564 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:52.174507 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:52.175309 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:52.176574 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:52.177056 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:52.177198 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:52.181031 (+ 3833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.1:49525: BlockingRecv error: recv got EOF from 127.6.61.1:49525 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 12 07:04:52 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:52 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:52.665279 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:52.656521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.2:59593)
0812 07:04:52.657122 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:52.657147 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:52.657316 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:52.657684 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:52.657688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:52.658201 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:52.658378 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:52.658388 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:52.659502 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:52.659510 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:52.660453 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:52.660465 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:52.661225 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:52.661231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:52.661414 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:52.662084 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:52.662103 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:52.664884 (+ 2781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":311,"threads_started":1}
W20260812 07:04:52.665390 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:52.656723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:59593 (local address 127.6.61.62:34289)
0812 07:04:52.656986 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:52.656995 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:52.657240 (+ 245us) server_negotiation.cc:407] Connection header received
0812 07:04:52.657464 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:52.657475 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:52.657538 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:52.657659 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:52.658531 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:52.659364 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:52.660610 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:52.661063 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:52.661200 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:52.665098 (+ 3898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.2:59593: BlockingRecv error: recv got EOF from 127.6.61.2:59593 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Aug 12 07:04:53 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:53 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:53.161793 17272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:53.153310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.3:50873)
0812 07:04:53.153921 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:53.153939 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:53.154051 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:53.154371 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:53.154374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:53.154787 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:53.154957 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:53.154967 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:53.155943 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:53.155946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:53.156867 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:53.156878 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:53.157562 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:53.157566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:53.157684 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:53.158423 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:53.158440 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:53.161431 (+ 2991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":282,"threads_started":1}
W20260812 07:04:53.161986 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:53.153385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:50873 (local address 127.6.61.62:34289)
0812 07:04:53.153558 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:53.153564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:53.154018 (+ 454us) server_negotiation.cc:407] Connection header received
0812 07:04:53.154148 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:53.154152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:53.154217 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:53.154300 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:53.155083 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:53.155805 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:53.156988 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:53.157437 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:53.157531 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:53.161717 (+ 4186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.3:50873: BlockingRecv error: recv got EOF from 127.6.61.3:50873 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 12 07:04:53 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:53 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:53.191110 17273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:53.182210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.1:34275)
0812 07:04:53.182808 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:53.182824 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:53.182992 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:53.183351 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:53.183357 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:53.183789 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:53.183945 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:53.183954 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:53.184999 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:53.185004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:53.185954 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:53.185967 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:53.186989 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:53.186995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:53.187204 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:53.187895 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:53.187914 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:53.190892 (+ 2978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":361,"threads_started":1}
W20260812 07:04:53.191484 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:53.182286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:34275 (local address 127.6.61.62:34289)
0812 07:04:53.182550 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:53.182558 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:53.182919 (+ 361us) server_negotiation.cc:407] Connection header received
0812 07:04:53.183128 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:53.183132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:53.183195 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:53.183300 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:53.184095 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:53.184840 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:53.186109 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:53.186817 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:53.186967 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:53.191095 (+ 4128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.1:34275: BlockingRecv error: recv got EOF from 127.6.61.1:34275 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 12 07:04:53 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:53 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:53.675500 17274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:53.666359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.2:37371)
0812 07:04:53.666896 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:53.666913 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:53.667079 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:53.667456 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:53.667461 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:53.667913 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:53.668091 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:53.668100 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:53.669127 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:53.669133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:53.670211 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:53.670224 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:53.670980 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:53.670999 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:53.671186 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:53.671830 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:53.671849 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:53.675202 (+ 3353us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":300,"threads_started":1}
W20260812 07:04:53.675781 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:53.666404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:37371 (local address 127.6.61.62:34289)
0812 07:04:53.666619 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:53.666629 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:53.667002 (+ 373us) server_negotiation.cc:407] Connection header received
0812 07:04:53.667214 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:53.667218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:53.667298 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:53.667422 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:53.668245 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:53.668976 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:53.670377 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:53.670806 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:53.670948 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:53.675532 (+ 4584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.2:37371: BlockingRecv error: recv got EOF from 127.6.61.2:37371 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 12 07:04:54 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:54 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:54.171996 17275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:54.162957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.3:51131)
0812 07:04:54.163552 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:54.163568 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:54.163687 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:54.164015 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:54.164018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:54.164428 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:54.164587 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:54.164596 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:54.166182 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:54.166190 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:54.167216 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:54.167232 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:54.168016 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:54.168022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:54.168202 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:54.168815 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:54.168834 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:54.171766 (+ 2932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":347,"threads_started":1}
W20260812 07:04:54.172220 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:54.163064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:51131 (local address 127.6.61.62:34289)
0812 07:04:54.163298 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:54.163303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:54.163650 (+ 347us) server_negotiation.cc:407] Connection header received
0812 07:04:54.163794 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:54.163798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:54.163867 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:54.163964 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:54.164751 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:54.166009 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:54.167384 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:54.167836 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:54.167982 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:54.171967 (+ 3985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.3:51131: BlockingRecv error: recv got EOF from 127.6.61.3:51131 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 12 07:04:54 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:54 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:54.200870 17276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:54.191995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.1:44335)
0812 07:04:54.192614 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:54.192630 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:54.192804 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:54.193153 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:54.193157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:54.193589 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:54.193777 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:54.193787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:54.194991 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:54.194997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:54.196007 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:54.196021 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:54.196792 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:54.196798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:54.196970 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:54.197585 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:54.197603 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:54.200458 (+ 2855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":265,"threads_started":1}
W20260812 07:04:54.200893 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:54.192090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:44335 (local address 127.6.61.62:34289)
0812 07:04:54.192297 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:54.192307 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:54.192728 (+ 421us) server_negotiation.cc:407] Connection header received
0812 07:04:54.192924 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:54.192929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:54.193013 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:54.193117 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:54.194046 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:54.194838 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:54.196173 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:54.196629 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:54.196771 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:54.200668 (+ 3897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.1:44335: BlockingRecv error: recv got EOF from 127.6.61.1:44335 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 12 07:04:54 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:54 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:54.686297 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:54.676675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.2:54593)
0812 07:04:54.677506 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:54.677533 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:54.677703 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:54.678161 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:54.678166 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:54.678792 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:54.678994 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:54.679005 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:54.680023 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:54.680029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:54.681017 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:54.681029 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:54.681787 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:54.681794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:54.682004 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:54.682745 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:54.682762 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:54.685930 (+ 3168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":451,"threads_started":1}
W20260812 07:04:54.686424 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:54.676951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:54593 (local address 127.6.61.62:34289)
0812 07:04:54.677184 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:54.677194 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:54.677621 (+ 427us) server_negotiation.cc:407] Connection header received
0812 07:04:54.677851 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:54.677856 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:54.677985 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:54.678102 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:54.679150 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:54.679879 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:54.681193 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:54.681623 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:54.681751 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:54.686156 (+ 4405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.2:54593: BlockingRecv error: recv got EOF from 127.6.61.2:54593 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 12 07:04:55 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:55 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:55.182046 17278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:55.173084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.3:54321)
0812 07:04:55.173736 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:55.173755 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:55.173949 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:55.174258 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:55.174262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:55.174669 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:55.174862 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:55.174871 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.175858 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.175862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:55.176772 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:55.176783 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.177464 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.177468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:55.177576 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:55.178309 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:55.178340 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:55.181630 (+ 3290us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":279,"threads_started":1}
W20260812 07:04:55.182125 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:55.173142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:54321 (local address 127.6.61.62:34289)
0812 07:04:55.173381 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:55.173391 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:55.173864 (+ 473us) server_negotiation.cc:407] Connection header received
0812 07:04:55.174063 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:55.174067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:55.174131 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:55.174212 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:55.174989 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.175736 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.176900 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.177333 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.177423 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:55.181873 (+ 4450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.3:54321: BlockingRecv error: recv got EOF from 127.6.61.3:54321 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 12 07:04:55 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:55 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:55.210767 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:55.202078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.1:37753)
0812 07:04:55.202745 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:55.202761 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:55.202931 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:55.203342 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:55.203347 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:55.203782 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:55.204018 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:55.204030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.205110 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.205114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:55.206206 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:55.206219 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.206946 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.206951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:55.207073 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:55.207696 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:55.207715 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:55.210535 (+ 2820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":382,"threads_started":1}
W20260812 07:04:55.211076 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:55.202187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:37753 (local address 127.6.61.62:34289)
0812 07:04:55.202361 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:55.202365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:55.202854 (+ 489us) server_negotiation.cc:407] Connection header received
0812 07:04:55.203075 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:55.203080 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:55.203161 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:55.203275 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:55.204186 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.204987 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.206331 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.206788 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.206886 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:55.210772 (+ 3886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.1:37753: BlockingRecv error: recv got EOF from 127.6.61.1:37753 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 12 07:04:55 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:55 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:55.696350 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:55.687474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.2:34231)
0812 07:04:55.688147 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:55.688170 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:55.688315 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:55.688819 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:55.688824 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:55.689249 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:55.689408 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:55.689416 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.690670 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.690676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:55.691625 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:55.691636 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.692399 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.692407 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:55.692578 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:55.693244 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:55.693265 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:55.696115 (+ 2850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":372,"threads_started":1}
W20260812 07:04:55.696674 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:55.687694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:34231 (local address 127.6.61.62:34289)
0812 07:04:55.688183 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:55.688190 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:55.688242 (+ 52us) server_negotiation.cc:407] Connection header received
0812 07:04:55.688472 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:55.688478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:55.688635 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:55.688744 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:55.689569 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.690523 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.691783 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.692197 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.692337 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:55.696347 (+ 4010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.2:34231: BlockingRecv error: recv got EOF from 127.6.61.2:34231 (error 108)
Metrics: {"server-negotiator.queue_time_us":347}
Aug 12 07:04:56 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:56 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:56.191226 17281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:56.183034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.3:53337)
0812 07:04:56.183557 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:56.183573 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:56.183686 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:56.183986 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:56.183990 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:56.184390 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:56.184549 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:56.184558 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.185511 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.185514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:56.186465 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:56.186476 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.187188 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.187192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:56.187313 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:56.187972 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:56.187990 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:56.190997 (+ 3007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":294,"threads_started":1}
W20260812 07:04:56.191486 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:56.183080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:53337 (local address 127.6.61.62:34289)
0812 07:04:56.183254 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:56.183259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:56.183646 (+ 387us) server_negotiation.cc:407] Connection header received
0812 07:04:56.183784 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:56.183788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:56.183860 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:56.183942 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:56.184675 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.185391 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.186591 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.187033 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.187126 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:56.191215 (+ 4089us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.3:53337: BlockingRecv error: recv got EOF from 127.6.61.3:53337 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 12 07:04:56 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:56 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:56.220981 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:56.211734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.1:37077)
0812 07:04:56.212289 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:56.212306 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:56.212485 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:56.212844 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:56.212849 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:56.213295 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:56.213478 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:56.213488 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.214650 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.214655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:56.215648 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:56.215664 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.216377 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.216383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:56.216580 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:56.217321 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:56.217347 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:56.220621 (+ 3274us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":282,"threads_started":1}
W20260812 07:04:56.221107 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:56.211804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:37077 (local address 127.6.61.62:34289)
0812 07:04:56.212046 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:56.212053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:56.212393 (+ 340us) server_negotiation.cc:407] Connection header received
0812 07:04:56.212634 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:56.212638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:56.212709 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:56.212811 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:56.213626 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.214511 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.215790 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.216211 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.216347 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:56.220861 (+ 4514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.1:37077: BlockingRecv error: recv got EOF from 127.6.61.1:37077 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 12 07:04:56 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:56 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:56.706449 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:56.697393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.2:60135)
0812 07:04:56.697982 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:56.697998 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:56.698124 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:56.698518 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:56.698522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:56.698970 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:56.699142 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:56.699151 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.700196 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.700200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:56.701130 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:56.701142 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.701921 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.701926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:56.702056 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:56.702644 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:56.702663 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:56.706185 (+ 3522us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":299,"threads_started":1}
W20260812 07:04:56.706656 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:56.697477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:60135 (local address 127.6.61.62:34289)
0812 07:04:56.697650 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:56.697655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:56.698091 (+ 436us) server_negotiation.cc:407] Connection header received
0812 07:04:56.698231 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:56.698235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:56.698318 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:56.698431 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:56.699278 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.700070 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.701259 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.701718 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.701847 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:56.706389 (+ 4542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.2:60135: BlockingRecv error: recv got EOF from 127.6.61.2:60135 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 12 07:04:57 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:57 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:57.204959 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:57.192193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.3:57769)
0812 07:04:57.192732 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:57.192749 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:57.192880 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:57.193207 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:57.193211 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:57.193634 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:57.193913 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:57.193927 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:57.194921 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:57.194925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:57.195996 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:57.196010 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:57.196778 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:57.196783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:57.196930 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:57.201527 (+ 4597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:57.201548 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:57.204707 (+ 3159us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":287,"threads_started":1}
W20260812 07:04:57.205194 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:57.192227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:57769 (local address 127.6.61.62:34289)
0812 07:04:57.192464 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:57.192473 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:57.192837 (+ 364us) server_negotiation.cc:407] Connection header received
0812 07:04:57.192973 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:57.192977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:57.193042 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:57.193123 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:57.194047 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:57.194776 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:57.196133 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:57.196632 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:57.196731 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:57.204969 (+ 8238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.3:57769: BlockingRecv error: recv got EOF from 127.6.61.3:57769 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 12 07:04:57 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:57 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:57.231575 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:57.222165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.1:46945)
0812 07:04:57.222998 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:57.223021 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:57.223194 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:57.223656 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:57.223660 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:57.224209 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:57.224406 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:57.224417 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:57.225461 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:57.225465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:57.226468 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:57.226481 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:57.227164 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:57.227169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:57.227291 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:57.227966 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:57.228000 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:57.231312 (+ 3312us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":522,"threads_started":1}
W20260812 07:04:57.231822 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:57.222376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:46945 (local address 127.6.61.62:34289)
0812 07:04:57.222595 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:57.222602 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:57.223115 (+ 513us) server_negotiation.cc:407] Connection header received
0812 07:04:57.223372 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:57.223377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:57.223461 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:57.223623 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:57.224556 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:57.225330 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:57.226612 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:57.227024 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:57.227119 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:57.231544 (+ 4425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.1:46945: BlockingRecv error: recv got EOF from 127.6.61.1:46945 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 12 07:04:57 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:57 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:57.715920 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:57.707405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.2:35257)
0812 07:04:57.707953 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:57.707970 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:57.708082 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:57.708392 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:57.708395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:57.708798 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:57.708950 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:57.708958 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:57.710100 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:57.710103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:57.711019 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:57.711030 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:57.711715 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:57.711720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:57.711849 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:57.712414 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:57.712434 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:57.715654 (+ 3220us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":322,"threads_started":1}
W20260812 07:04:57.716094 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:57.707528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:35257 (local address 127.6.61.62:34289)
0812 07:04:57.707706 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:57.707711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:57.708054 (+ 343us) server_negotiation.cc:407] Connection header received
0812 07:04:57.708193 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:57.708197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:57.708267 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:57.708360 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:57.709102 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:57.709947 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:57.711146 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:57.711571 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:57.711663 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:57.715868 (+ 4205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.2:35257: BlockingRecv error: recv got EOF from 127.6.61.2:35257 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 12 07:04:58 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:58 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:58.215071 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:58.206139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.3:60491)
0812 07:04:58.206766 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:58.206783 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:58.206915 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:58.207239 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:58.207243 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:58.207748 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:58.207940 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:58.207951 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:58.209207 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:58.209212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:58.210330 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:58.210344 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:58.211046 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:58.211051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:58.211170 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:58.211761 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:58.211778 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:58.214807 (+ 3029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":384,"threads_started":1}
W20260812 07:04:58.215384 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:58.206433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:60491 (local address 127.6.61.62:34289)
0812 07:04:58.206676 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:58.206683 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:58.206882 (+ 199us) server_negotiation.cc:407] Connection header received
0812 07:04:58.207016 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:58.207020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:58.207087 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:58.207214 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:58.208098 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:58.209028 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:58.210475 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:58.210911 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:58.211008 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:58.215085 (+ 4077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.3:60491: BlockingRecv error: recv got EOF from 127.6.61.3:60491 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
Aug 12 07:04:58 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:58 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:58.241323 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:58.232652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.1:52867)
0812 07:04:58.233224 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:58.233242 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:58.233357 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:58.233751 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:58.233756 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:58.234237 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:58.234443 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:58.234453 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:58.235529 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:58.235536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:58.236465 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:58.236478 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:58.237226 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:58.237232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:58.237430 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:58.238071 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:58.238100 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:58.241027 (+ 2927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":293,"threads_started":1}
W20260812 07:04:58.241518 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:58.232723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:52867 (local address 127.6.61.62:34289)
0812 07:04:58.232991 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:58.232998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:58.233313 (+ 315us) server_negotiation.cc:407] Connection header received
0812 07:04:58.233481 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:58.233487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:58.233578 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:58.233690 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:58.234602 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:58.235386 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:58.236617 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:58.237062 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:58.237201 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:58.241243 (+ 4042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.1:52867: BlockingRecv error: recv got EOF from 127.6.61.1:52867 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 12 07:04:58 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:58 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:58.725503 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:58.716906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.2:60647)
0812 07:04:58.717633 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:58.717650 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:58.717866 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:58.718289 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:58.718294 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:58.718875 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:58.719151 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:58.719164 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:58.720171 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:58.720175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:58.721086 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:58.721098 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:58.721759 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:58.721763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:58.721902 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:58.722489 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:58.722506 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:58.725236 (+ 2730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":305,"threads_started":1}
W20260812 07:04:58.725672 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:58.717146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:60647 (local address 127.6.61.62:34289)
0812 07:04:58.717377 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:58.717383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:58.717742 (+ 359us) server_negotiation.cc:407] Connection header received
0812 07:04:58.718025 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:58.718030 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:58.718131 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:58.718215 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:58.719282 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:58.720038 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:58.721207 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:58.721604 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:58.721695 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:58.725436 (+ 3741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.2:60647: BlockingRecv error: recv got EOF from 127.6.61.2:60647 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 12 07:04:59 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:59 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:59.224699 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:59.215987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.3:50203)
0812 07:04:59.216563 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:59.216580 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:59.216736 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:59.217126 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:59.217131 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:59.217583 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:59.217801 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:59.217884 (+ 83us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:59.218878 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:59.218882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:59.219771 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:59.219782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:59.220484 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:59.220488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:59.220607 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:59.221211 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:59.221228 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:59.224343 (+ 3115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":301,"threads_started":1}
W20260812 07:04:59.224897 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:59.216062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:50203 (local address 127.6.61.62:34289)
0812 07:04:59.216344 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:59.216350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:59.216663 (+ 313us) server_negotiation.cc:407] Connection header received
0812 07:04:59.216900 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:59.216904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:59.216969 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:59.217052 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:59.218020 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:59.218750 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:59.219913 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:59.220350 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:59.220445 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:59.224612 (+ 4167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.3:50203: BlockingRecv error: recv got EOF from 127.6.61.3:50203 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 12 07:04:59 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:59 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:59.251893 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:59.242375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.1:45551)
0812 07:04:59.242992 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:59.243009 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:59.243127 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:59.243441 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:59.243445 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:59.244085 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:59.244313 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:59.244325 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:59.245429 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:59.245434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:59.246493 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:59.246508 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:59.247279 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:59.247284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:59.247430 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:59.248056 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:59.248074 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:59.251481 (+ 3407us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":329,"threads_started":1}
W20260812 07:04:59.252020 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:59.242436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:45551 (local address 127.6.61.62:34289)
0812 07:04:59.242641 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:59.242646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:59.243082 (+ 436us) server_negotiation.cc:407] Connection header received
0812 07:04:59.243227 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:59.243231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:59.243296 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:59.243542 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:59.244486 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:59.245280 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:59.246641 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:59.247122 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:59.247222 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:59.251736 (+ 4514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.1:45551: BlockingRecv error: recv got EOF from 127.6.61.1:45551 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 12 07:04:59 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:04:59 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:04:59.739140 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:59.726666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.2:34583)
0812 07:04:59.727248 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:59.727272 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:59.727400 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:59.727832 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:59.727839 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:59.728419 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:59.728675 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:59.728691 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:59.730251 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:59.730260 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:59.732104 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:59.732125 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:59.733334 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:59.733343 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:59.733584 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:59.734704 (+ 1120us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:59.734728 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:59.738784 (+ 4056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":330,"threads_started":1}
W20260812 07:04:59.739279 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:59.726739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:34583 (local address 127.6.61.62:34289)
0812 07:04:59.726922 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:59.726927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:59.727338 (+ 411us) server_negotiation.cc:407] Connection header received
0812 07:04:59.727513 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:59.727517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:59.727616 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:59.727749 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:59.728860 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:59.730071 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:59.732281 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:59.732907 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:59.733072 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:59.739015 (+ 5943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.2:34583: BlockingRecv error: recv got EOF from 127.6.61.2:34583 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 12 07:05:00 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:05:00 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:05:00.234953 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:00.225799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.3:49125)
0812 07:05:00.226366 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:00.226383 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:00.226499 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:00.226791 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:00.226794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:00.227192 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:05:00.227354 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:00.227364 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:00.228490 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:00.228494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:00.229548 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:00.229561 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:00.230344 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:00.230349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:00.230478 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:00.231096 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:05:00.231114 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:05:00.234706 (+ 3592us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":286,"threads_started":1}
W20260812 07:05:00.235190 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:05:00.225928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:49125 (local address 127.6.61.62:34289)
0812 07:05:00.226193 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:00.226200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:00.226463 (+ 263us) server_negotiation.cc:407] Connection header received
0812 07:05:00.226597 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:00.226601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:00.226665 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:00.226746 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:00.227480 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:00.228315 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:00.229673 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:00.230102 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:00.230265 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:00.234908 (+ 4643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.3:49125: BlockingRecv error: recv got EOF from 127.6.61.3:49125 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 12 07:05:00 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:05:00 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:05:00.261920 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:00.252847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.1:56953)
0812 07:05:00.253475 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:00.253508 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:00.253623 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:00.253965 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:00.253968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:00.254371 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:05:00.254558 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:00.254567 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:00.255592 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:00.255596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:00.256889 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:00.256906 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:00.257980 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:00.257985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:00.258118 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:00.258690 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:05:00.258708 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:05:00.261525 (+ 2817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":313,"threads_started":1}
W20260812 07:05:00.262058 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:05:00.252942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:56953 (local address 127.6.61.62:34289)
0812 07:05:00.253164 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:00.253170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:00.253590 (+ 420us) server_negotiation.cc:407] Connection header received
0812 07:05:00.253709 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:00.253712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:00.253775 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:00.253950 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:00.254685 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:00.255455 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:00.257068 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:00.257745 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:00.257878 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:00.261766 (+ 3888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.1:56953: BlockingRecv error: recv got EOF from 127.6.61.1:56953 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 12 07:05:00 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:05:00 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:05:00.749748 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:00.740350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.2:60755)
0812 07:05:00.741017 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:00.741034 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:00.741196 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:00.741559 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:00.741563 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:00.742062 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:05:00.742251 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:00.742268 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:00.743342 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:00.743348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:00.744274 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:00.744285 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:00.745089 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:00.745095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:00.745282 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:00.746108 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:05:00.746138 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:05:00.749377 (+ 3239us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":402,"threads_started":1}
W20260812 07:05:00.749866 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:05:00.740554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:60755 (local address 127.6.61.62:34289)
0812 07:05:00.740844 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:00.740851 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:00.741128 (+ 277us) server_negotiation.cc:407] Connection header received
0812 07:05:00.741348 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:00.741353 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:00.741416 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:00.741530 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:00.742409 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:00.743158 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:00.744423 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:00.744884 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:00.745078 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:00.749574 (+ 4496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.2:60755: BlockingRecv error: recv got EOF from 127.6.61.2:60755 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Aug 12 07:05:01 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:05:01 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:05:01.244836 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:01.236139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.3:37395)
0812 07:05:01.236669 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:01.236703 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:01.236827 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:01.237175 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:01.237179 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:01.237598 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:05:01.237792 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:01.237801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:01.238956 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:01.238962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:01.239892 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:01.239904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:01.240659 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:01.240665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:01.240843 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:01.241539 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:05:01.241558 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:05:01.244587 (+ 3029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":286,"threads_started":1}
W20260812 07:05:01.245035 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:05:01.236189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:37395 (local address 127.6.61.62:34289)
0812 07:05:01.236453 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:01.236461 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:01.236791 (+ 330us) server_negotiation.cc:407] Connection header received
0812 07:05:01.236936 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:01.236939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:01.237007 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:01.237142 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:01.238015 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:01.238808 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:01.240041 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:01.240469 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:01.240608 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:01.244801 (+ 4193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.3:37395: BlockingRecv error: recv got EOF from 127.6.61.3:37395 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 12 07:05:01 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:05:01 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:05:01.272099 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:01.263091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.1:56969)
0812 07:05:01.263663 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:01.263681 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:01.263804 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:01.264100 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:01.264103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:01.264525 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:05:01.264682 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:01.264691 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:01.266067 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:01.266072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:01.267154 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:01.267169 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:01.267913 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:01.267918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:01.268043 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:01.268667 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:05:01.268687 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:05:01.271807 (+ 3120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":319,"threads_started":1}
W20260812 07:05:01.272302 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:05:01.263188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:56969 (local address 127.6.61.62:34289)
0812 07:05:01.263472 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:01.263481 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:01.263770 (+ 289us) server_negotiation.cc:407] Connection header received
0812 07:05:01.263893 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:01.263897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:01.263966 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:01.264053 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:01.264866 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:01.265912 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:01.267299 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:01.267746 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:01.267865 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:01.271993 (+ 4128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.1:56969: BlockingRecv error: recv got EOF from 127.6.61.1:56969 (error 108)
Metrics: {"server-negotiator.queue_time_us":157}
Aug 12 07:05:01 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:05:01 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:05:01.760221 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:01.750882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.2:38537)
0812 07:05:01.751452 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:01.751470 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:01.751596 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:01.751990 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:01.751993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:01.752354 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:05:01.752573 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:01.752603 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:01.753782 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:01.753786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:01.754812 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:01.754825 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:01.755598 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:01.755602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:01.755727 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:01.756376 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:05:01.756399 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:05:01.759885 (+ 3486us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":265,"threads_started":1}
W20260812 07:05:01.760344 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:05:01.751032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:38537 (local address 127.6.61.62:34289)
0812 07:05:01.751308 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:01.751315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:01.751553 (+ 238us) server_negotiation.cc:407] Connection header received
0812 07:05:01.751749 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:01.751753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:01.751839 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:01.751949 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:01.752795 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:01.753629 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:01.754958 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:01.755466 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:01.755566 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:01.760100 (+ 4534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.2:38537: BlockingRecv error: recv got EOF from 127.6.61.2:38537 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Aug 12 07:05:02 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:05:02 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.3@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:05:02.260242 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:02.246238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.3:39037)
0812 07:05:02.246811 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:02.246828 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:02.246949 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:02.247289 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:02.247292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:02.247704 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:05:02.247889 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:02.247897 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:02.249077 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:02.249082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:02.250143 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:02.250173 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:02.251189 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:02.251195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:02.251447 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:02.256850 (+ 5403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:05:02.256872 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:05:02.259981 (+ 3109us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":300,"threads_started":1}
W20260812 07:05:02.260551 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:05:02.246354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:39037 (local address 127.6.61.62:34289)
0812 07:05:02.246595 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:02.246601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:02.246950 (+ 349us) server_negotiation.cc:407] Connection header received
0812 07:05:02.247045 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:02.247049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:02.247142 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:02.247255 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:02.248049 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:02.248903 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:02.250319 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:02.251010 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:02.251170 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:02.260237 (+ 9067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.3:39037: BlockingRecv error: recv got EOF from 127.6.61.3:39037 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 12 07:05:02 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:05:02 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.1@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:05:02.282757 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:02.273498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.1:56665)
0812 07:05:02.274073 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:02.274090 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:02.274263 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:02.274672 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:02.274677 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:02.275184 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:05:02.275450 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:02.275462 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:02.276751 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:02.276756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:02.277900 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:02.277924 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:02.278724 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:02.278728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:02.278873 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:02.279538 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:05:02.279557 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:05:02.282501 (+ 2944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":343,"threads_started":1}
W20260812 07:05:02.282990 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:05:02.273582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:56665 (local address 127.6.61.62:34289)
0812 07:05:02.273861 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:02.273871 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:02.274181 (+ 310us) server_negotiation.cc:407] Connection header received
0812 07:05:02.274416 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:02.274421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:02.274525 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:02.274638 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:02.275639 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:02.276623 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:02.278071 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:02.278591 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:02.278706 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:02.282702 (+ 3996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.1:56665: BlockingRecv error: recv got EOF from 127.6.61.1:56665 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 12 07:05:02 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:05:02 dist-test-slave-69rg krb5kdc[16066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.61.2@KRBTEST.COM for oryx/127.6.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:05:02.770470 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:02.761477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34289 (local address 127.6.61.2:43439)
0812 07:05:02.762143 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:02.762164 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:02.762323 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:02.762714 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:02.762718 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:02.763167 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:05:02.763351 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:02.763360 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:02.764481 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:02.764488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:02.765432 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:02.765444 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:02.766286 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:02.766293 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:02.766471 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:02.767100 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:05:02.767120 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:05:02.770094 (+ 2974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.61.62:34289: Server oryx/127.6.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":308,"threads_started":1}
W20260812 07:05:02.770578 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:05:02.761664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:43439 (local address 127.6.61.62:34289)
0812 07:05:02.761936 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:02.761942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:02.762283 (+ 341us) server_negotiation.cc:407] Connection header received
0812 07:05:02.762458 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:02.762464 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:02.762554 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:02.762684 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:02.763506 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:02.764334 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:02.765579 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:02.766116 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:02.766266 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:02.770313 (+ 4047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.61.2:43439: BlockingRecv error: recv got EOF from 127.6.61.2:43439 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
I20260812 07:05:03.071194 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 16756
I20260812 07:05:03.085074 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:03.072448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38948 (local address 127.6.61.1:38819)
0812 07:05:03.073070 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:03.073078 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:03.073107 (+ 29us) server_negotiation.cc:407] Connection header received
0812 07:05:03.073182 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:03.073189 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:03.073411 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:03.073595 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:03.075313 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:03.076401 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:03.077688 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:03.078421 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:03.078531 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:03.081451 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:03.081461 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:03.081477 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:05:03.081517 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:03.084546 (+ 3029us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.61.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0812 07:05:03.084831 (+ 285us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38948: Request ticket server kudu/127.6.61.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":348,"threads_started":1}
W20260812 07:05:03.085227 17306 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38948: Request ticket server kudu/127.6.61.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260812 07:05:03.085520 6388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 16756: Not authorized: Client connection negotiation failed: client connection to 127.6.61.1:38819: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.61.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260812 07:05:03.085776 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 16756
I20260812 07:05:03.105175 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 16944
I20260812 07:05:03.119966 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:03.106371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57310 (local address 127.6.61.2:35135)
0812 07:05:03.107039 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:03.107047 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:03.107082 (+ 35us) server_negotiation.cc:407] Connection header received
0812 07:05:03.107194 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:03.107201 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:03.107456 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:03.107666 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:03.109250 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:03.110594 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:03.112012 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:03.112657 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:03.112818 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:03.115976 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:03.115986 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:03.116003 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:05:03.116053 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:03.119148 (+ 3095us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.61.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0812 07:05:03.119534 (+ 386us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57310: Request ticket server kudu/127.6.61.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":325,"threads_started":1}
W20260812 07:05:03.120057 6388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 16944: Not authorized: Client connection negotiation failed: client connection to 127.6.61.2:35135: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.61.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260812 07:05:03.120119 17307 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57310: Request ticket server kudu/127.6.61.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260812 07:05:03.120415 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 16944
I20260812 07:05:03.143631 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 17097
I20260812 07:05:03.157521 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:03.144842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43810 (local address 127.6.61.3:45341)
0812 07:05:03.145523 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:03.145531 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:03.145560 (+ 29us) server_negotiation.cc:407] Connection header received
0812 07:05:03.145627 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:03.145634 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:03.145925 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:03.146098 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:03.147497 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:03.148658 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:03.149980 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:03.150690 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:03.150815 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:03.153971 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:03.153981 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:03.153997 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:05:03.154073 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:03.156880 (+ 2807us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.61.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0812 07:05:03.157221 (+ 341us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43810: Request ticket server kudu/127.6.61.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":365,"threads_started":1}
W20260812 07:05:03.157641 17308 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43810: Request ticket server kudu/127.6.61.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260812 07:05:03.157649 6388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 17097: Not authorized: Client connection negotiation failed: client connection to 127.6.61.3:45341: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.61.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260812 07:05:03.158066 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 17097
I20260812 07:05:03.180567 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 16673
I20260812 07:05:03.182106 16748 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:05:03.301724 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 16673
2026-08-12T07:05:03Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21536 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-12T07:05:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:05:03Z Disabled control of system clock
I20260812 07:05:03.397069 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:42145
--webserver_interface=127.6.61.62
--webserver_port=0
--builtin_ntp_servers=127.6.61.20:45243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:42145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 07:05:03.740531 17315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:05:03.740864 17315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:05:03.740980 17315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:05:03.748306 17315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:05:03.748425 17315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:05:03.748464 17315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:05:03.748517 17315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:05:03.748565 17315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:05:03.757771 17315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:45243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:42145
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:42145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:05:03.759169 17315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:05:03.760955 17315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:05:03.773988 17323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:03.773962 17322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:03.774046 17325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:05:03.774711 17315 server_base.cc:1034] running on GCE node
I20260812 07:05:03.775637 17315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:05:03.777276 17315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:05:03.778537 17315 hybrid_clock.cc:648] HybridClock initialized: now 1786518303778474 us; error 62 us; skew 500 ppm
I20260812 07:05:03.778970 17315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 07:05:03.781605 17315 webserver.cc:460] Webserver started at http://127.6.61.62:35459/ using document root <none> and password file <none>
I20260812 07:05:03.782449 17315 fs_manager.cc:359] Metadata directory not provided
I20260812 07:05:03.782594 17315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:05:03.782949 17315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:05:03.786231 17315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "287f5cea369145879a2ceb51142cc5be"
format_stamp: "Formatted at 2026-08-12 07:05:03 on dist-test-slave-69rg"
server_key: "ea059825dcc66ae0ca2ec3053b51659a"
server_key_iv: "7e915cacca88de92e64859681e2240da"
server_key_version: "encryptionkey@0"
I20260812 07:05:03.787225 17315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "287f5cea369145879a2ceb51142cc5be"
format_stamp: "Formatted at 2026-08-12 07:05:03 on dist-test-slave-69rg"
server_key: "ea059825dcc66ae0ca2ec3053b51659a"
server_key_iv: "7e915cacca88de92e64859681e2240da"
server_key_version: "encryptionkey@0"
I20260812 07:05:03.792563 17315 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 07:05:03.796288 17331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:03.797741 17315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 07:05:03.797988 17315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "287f5cea369145879a2ceb51142cc5be"
format_stamp: "Formatted at 2026-08-12 07:05:03 on dist-test-slave-69rg"
server_key: "ea059825dcc66ae0ca2ec3053b51659a"
server_key_iv: "7e915cacca88de92e64859681e2240da"
server_key_version: "encryptionkey@0"
I20260812 07:05:03.798266 17315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:05:03.823864 17315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:05:03.824944 17315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:05:03.825309 17315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:05:03.847500 17315 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:42145
I20260812 07:05:03.847535 17389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:42145 every 8 connection(s)
I20260812 07:05:03.849082 17315 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:05:03.853256 17390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:05:03.855136 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 17315
I20260812 07:05:03.855383 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:05:03.856087 6388 external_mini_cluster.cc:1428] Setting key c02fb20ff6ec40cae004e92f117b4fb0
I20260812 07:05:03.865556 17390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 287f5cea369145879a2ceb51142cc5be: Bootstrap starting.
I20260812 07:05:03.869370 17390 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 287f5cea369145879a2ceb51142cc5be: Neither blocks nor log segments found. Creating new log.
I20260812 07:05:03.870641 17390 log.cc:824] T 00000000000000000000000000000000 P 287f5cea369145879a2ceb51142cc5be: Log is configured to *not* fsync() on all Append() calls
I20260812 07:05:03.871325 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:03.858533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35852 (local address 127.6.61.62:42145)
0812 07:05:03.859632 (+ 1099us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:03.859643 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:03.859688 (+ 45us) server_negotiation.cc:407] Connection header received
0812 07:05:03.861363 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:03.861387 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:03.861938 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:03.862294 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:03.863018 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:03.864416 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:03.866355 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:03.867085 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:03.867292 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:03.868654 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:03.868662 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:03.868675 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:05:03.869214 (+ 539us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:03.869376 (+ 162us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:05:03.869402 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:05:03.869567 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:03.869811 (+ 244us) server_negotiation.cc:299] Negotiation successful
0812 07:05:03.870208 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":433,"threads_started":1}
I20260812 07:05:03.873874 17390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 287f5cea369145879a2ceb51142cc5be: No bootstrap required, opened a new log
I20260812 07:05:03.881457 17390 raft_consensus.cc:348] T 00000000000000000000000000000000 P 287f5cea369145879a2ceb51142cc5be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287f5cea369145879a2ceb51142cc5be" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 42145 } }
I20260812 07:05:03.881918 17390 raft_consensus.cc:374] T 00000000000000000000000000000000 P 287f5cea369145879a2ceb51142cc5be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:05:03.882042 17390 raft_consensus.cc:729] T 00000000000000000000000000000000 P 287f5cea369145879a2ceb51142cc5be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 287f5cea369145879a2ceb51142cc5be, State: Initialized, Role: FOLLOWER
I20260812 07:05:03.882611 17390 consensus_queue.cc:260] T 00000000000000000000000000000000 P 287f5cea369145879a2ceb51142cc5be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287f5cea369145879a2ceb51142cc5be" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 42145 } }
I20260812 07:05:03.882850 17390 raft_consensus.cc:388] T 00000000000000000000000000000000 P 287f5cea369145879a2ceb51142cc5be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:05:03.883000 17390 raft_consensus.cc:482] T 00000000000000000000000000000000 P 287f5cea369145879a2ceb51142cc5be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:05:03.883174 17390 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 287f5cea369145879a2ceb51142cc5be [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:05:03.885574 17390 raft_consensus.cc:504] T 00000000000000000000000000000000 P 287f5cea369145879a2ceb51142cc5be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287f5cea369145879a2ceb51142cc5be" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 42145 } }
I20260812 07:05:03.886178 17390 leader_election.cc:302] T 00000000000000000000000000000000 P 287f5cea369145879a2ceb51142cc5be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 287f5cea369145879a2ceb51142cc5be; no voters:
I20260812 07:05:03.886821 17390 leader_election.cc:288] T 00000000000000000000000000000000 P 287f5cea369145879a2ceb51142cc5be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:05:03.887007 17395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 287f5cea369145879a2ceb51142cc5be [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:05:03.887737 17395 raft_consensus.cc:686] T 00000000000000000000000000000000 P 287f5cea369145879a2ceb51142cc5be [term 1 LEADER]: Becoming Leader. State: Replica: 287f5cea369145879a2ceb51142cc5be, State: Running, Role: LEADER
I20260812 07:05:03.888367 17395 consensus_queue.cc:237] T 00000000000000000000000000000000 P 287f5cea369145879a2ceb51142cc5be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287f5cea369145879a2ceb51142cc5be" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 42145 } }
I20260812 07:05:03.888998 17390 sys_catalog.cc:564] T 00000000000000000000000000000000 P 287f5cea369145879a2ceb51142cc5be [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:05:03.891974 17397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 287f5cea369145879a2ceb51142cc5be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 287f5cea369145879a2ceb51142cc5be. Latest consensus state: current_term: 1 leader_uuid: "287f5cea369145879a2ceb51142cc5be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287f5cea369145879a2ceb51142cc5be" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 42145 } } }
I20260812 07:05:03.892410 17397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 287f5cea369145879a2ceb51142cc5be [sys.catalog]: This master's current role is: LEADER
I20260812 07:05:03.892198 17396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 287f5cea369145879a2ceb51142cc5be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "287f5cea369145879a2ceb51142cc5be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287f5cea369145879a2ceb51142cc5be" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 42145 } } }
I20260812 07:05:03.894337 17408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:05:03.896085 17396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 287f5cea369145879a2ceb51142cc5be [sys.catalog]: This master's current role is: LEADER
I20260812 07:05:03.901598 17408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:05:03.909781 17408 catalog_manager.cc:1347] Generated new cluster ID: eab485c4c03c465ba8964d38a508f987
I20260812 07:05:03.909945 17408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:05:03.944307 17408 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:05:03.945752 17408 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:05:03.961587 17408 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 287f5cea369145879a2ceb51142cc5be: Generated new TSK 0
I20260812 07:05:03.962544 17408 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:05:03.972193 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.1:0
--local_ip_for_outbound_sockets=127.6.61.1
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:42145
--builtin_ntp_servers=127.6.61.20:45243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260812 07:05:04.324178 17421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:05:04.324473 17421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:05:04.324623 17421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:05:04.332780 17421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:05:04.332895 17421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:05:04.333091 17421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.1
I20260812 07:05:04.342623 17421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:45243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:42145
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.1
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:05:04.343981 17421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:05:04.345956 17421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:05:04.359237 17427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:04.359198 17428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:04.359261 17430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:05:04.360363 17421 server_base.cc:1034] running on GCE node
I20260812 07:05:04.361076 17421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:05:04.362112 17421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:05:04.363345 17421 hybrid_clock.cc:648] HybridClock initialized: now 1786518304363296 us; error 33 us; skew 500 ppm
I20260812 07:05:04.363722 17421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 07:05:04.366168 17421 webserver.cc:460] Webserver started at http://127.6.61.1:43289/ using document root <none> and password file <none>
I20260812 07:05:04.366909 17421 fs_manager.cc:359] Metadata directory not provided
I20260812 07:05:04.367012 17421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:05:04.367408 17421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:05:04.370836 17421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/data/instance:
uuid: "cb4d9da79d3d45e49af0dac39b9df28c"
format_stamp: "Formatted at 2026-08-12 07:05:04 on dist-test-slave-69rg"
server_key: "4af9def51cb1829feaa3ead6bebfd007"
server_key_iv: "8e3438a19f4345156aca6dbfd615f509"
server_key_version: "encryptionkey@0"
I20260812 07:05:04.371723 17421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance:
uuid: "cb4d9da79d3d45e49af0dac39b9df28c"
format_stamp: "Formatted at 2026-08-12 07:05:04 on dist-test-slave-69rg"
server_key: "4af9def51cb1829feaa3ead6bebfd007"
server_key_iv: "8e3438a19f4345156aca6dbfd615f509"
server_key_version: "encryptionkey@0"
I20260812 07:05:04.377054 17421 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 07:05:04.380802 17436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:04.382367 17421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:05:04.382560 17421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/wal
uuid: "cb4d9da79d3d45e49af0dac39b9df28c"
format_stamp: "Formatted at 2026-08-12 07:05:04 on dist-test-slave-69rg"
server_key: "4af9def51cb1829feaa3ead6bebfd007"
server_key_iv: "8e3438a19f4345156aca6dbfd615f509"
server_key_version: "encryptionkey@0"
I20260812 07:05:04.382799 17421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:05:04.404101 17421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:05:04.404929 17421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:05:04.405180 17421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:05:04.406147 17421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:05:04.407698 17421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:05:04.407790 17421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:04.407882 17421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:05:04.407938 17421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:04.441592 17421 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.1:43327
I20260812 07:05:04.441627 17568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.1:43327 every 8 connection(s)
I20260812 07:05:04.442983 17421 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
I20260812 07:05:04.452894 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 17421
I20260812 07:05:04.453130 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance
I20260812 07:05:04.454149 6388 external_mini_cluster.cc:1428] Setting key 60d3f4df369ba8b5c089c0fc9495fa2d
I20260812 07:05:04.456444 17571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:04.445741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:52431 (local address 127.6.61.62:42145)
0812 07:05:04.446288 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:04.446292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:04.447430 (+ 1138us) server_negotiation.cc:407] Connection header received
0812 07:05:04.449253 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:04.449259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:04.449369 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:04.449497 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:04.450686 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:04.451418 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:04.453118 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:04.453801 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:04.453980 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:04.455550 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:04.455555 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:04.455558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:05:04.455577 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:04.455629 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:05:04.455641 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:05:04.455756 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:04.455988 (+ 232us) server_negotiation.cc:299] Negotiation successful
0812 07:05:04.456088 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":309,"threads_started":1}
I20260812 07:05:04.457067 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.2:0
--local_ip_for_outbound_sockets=127.6.61.2
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:42145
--builtin_ntp_servers=127.6.61.20:45243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 07:05:04.457608 17572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:04.446213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:42145 (local address 127.6.61.1:52431)
0812 07:05:04.447240 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:04.447286 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:04.448960 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:04.449747 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:04.449756 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:04.449901 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:05:04.450488 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:04.450506 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:04.451589 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:04.451594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:04.452906 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:04.452919 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:04.454067 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:04.454073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:04.454382 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:04.455179 (+ 797us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:05:04.455182 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:05:04.455224 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:05:04.455316 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:05:04.455758 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:05:04.455771 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:05:04.455782 (+ 11us) client_negotiation.cc:769] Sending connection context
0812 07:05:04.455976 (+ 194us) client_negotiation.cc:240] Negotiation successful
0812 07:05:04.456404 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":328,"threads_started":1}
I20260812 07:05:04.459478 17569 heartbeater.cc:347] Connected to a master server at 127.6.61.62:42145
I20260812 07:05:04.459817 17569 heartbeater.cc:464] Registering TS with master...
I20260812 07:05:04.460697 17569 heartbeater.cc:511] Master 127.6.61.62:42145 requested a full tablet report, sending...
I20260812 07:05:04.462918 17352 ts_manager.cc:194] Registered new tserver with Master: cb4d9da79d3d45e49af0dac39b9df28c (127.6.61.1:43327)
I20260812 07:05:04.464533 17352 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.61.1:52431
W20260812 07:05:04.803821 17573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:05:04.804134 17573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:05:04.804224 17573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:05:04.811908 17573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:05:04.812031 17573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:05:04.812137 17573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.2
I20260812 07:05:04.821657 17573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:45243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:42145
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.2
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:05:04.823060 17573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:05:04.824946 17573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:05:04.839093 17580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:04.839051 17581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:04.838976 17583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:05:04.840910 17573 server_base.cc:1034] running on GCE node
I20260812 07:05:04.841679 17573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:05:04.842762 17573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:05:04.844018 17573 hybrid_clock.cc:648] HybridClock initialized: now 1786518304843968 us; error 52 us; skew 500 ppm
I20260812 07:05:04.844414 17573 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 07:05:04.847021 17573 webserver.cc:460] Webserver started at http://127.6.61.2:43825/ using document root <none> and password file <none>
I20260812 07:05:04.847816 17573 fs_manager.cc:359] Metadata directory not provided
I20260812 07:05:04.847962 17573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:05:04.848323 17573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:05:04.851366 17573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/data/instance:
uuid: "4bf2c6ecdd814a79ade88b4bfcd478a6"
format_stamp: "Formatted at 2026-08-12 07:05:04 on dist-test-slave-69rg"
server_key: "66cc882fc6c8c6504acb5056dc6e21ae"
server_key_iv: "d9e254b40abdbd903d6504ba5e9f0eb4"
server_key_version: "encryptionkey@0"
I20260812 07:05:04.852249 17573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance:
uuid: "4bf2c6ecdd814a79ade88b4bfcd478a6"
format_stamp: "Formatted at 2026-08-12 07:05:04 on dist-test-slave-69rg"
server_key: "66cc882fc6c8c6504acb5056dc6e21ae"
server_key_iv: "d9e254b40abdbd903d6504ba5e9f0eb4"
server_key_version: "encryptionkey@0"
I20260812 07:05:04.858202 17573 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260812 07:05:04.862324 17589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:04.863932 17573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 07:05:04.864151 17573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/wal
uuid: "4bf2c6ecdd814a79ade88b4bfcd478a6"
format_stamp: "Formatted at 2026-08-12 07:05:04 on dist-test-slave-69rg"
server_key: "66cc882fc6c8c6504acb5056dc6e21ae"
server_key_iv: "d9e254b40abdbd903d6504ba5e9f0eb4"
server_key_version: "encryptionkey@0"
I20260812 07:05:04.864414 17573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:05:04.890123 17573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:05:04.891026 17573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:05:04.891309 17573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:05:04.892306 17573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:05:04.894040 17573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:05:04.894143 17573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:04.894229 17573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:05:04.894279 17573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:04.930410 17573 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.2:40381
I20260812 07:05:04.930505 17724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.2:40381 every 8 connection(s)
I20260812 07:05:04.932273 17573 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
I20260812 07:05:04.938264 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 17573
I20260812 07:05:04.938483 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance
I20260812 07:05:04.939141 6388 external_mini_cluster.cc:1428] Setting key 4ce6a205ece2ec7a60e17a7cf6440b84
I20260812 07:05:04.942615 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.3:0
--local_ip_for_outbound_sockets=127.6.61.3
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:42145
--builtin_ntp_servers=127.6.61.20:45243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 07:05:04.948851 17571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:04.935844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:40353 (local address 127.6.61.62:42145)
0812 07:05:04.936061 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:04.936067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:04.937552 (+ 1485us) server_negotiation.cc:407] Connection header received
0812 07:05:04.940485 (+ 2933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:04.940491 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:04.940577 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:04.940700 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:04.942190 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:04.943041 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:04.945670 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:04.946146 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:04.946287 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:04.948033 (+ 1746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:04.948038 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:04.948042 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:05:04.948063 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:04.948098 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:05:04.948111 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:05:04.948207 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:04.948498 (+ 291us) server_negotiation.cc:299] Negotiation successful
0812 07:05:04.948590 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260812 07:05:04.950438 17727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:04.936373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:42145 (local address 127.6.61.2:40353)
0812 07:05:04.937411 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:04.937450 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:04.940072 (+ 2622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:04.940968 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:04.940977 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:04.941087 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:05:04.941970 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:04.941998 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:04.943208 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:04.943217 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:04.945497 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:04.945518 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:04.946389 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:04.946399 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:04.946768 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:04.947614 (+ 846us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:05:04.947619 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:05:04.947678 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:05:04.947807 (+ 129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:05:04.948225 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:05:04.948238 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:05:04.948249 (+ 11us) client_negotiation.cc:769] Sending connection context
0812 07:05:04.948448 (+ 199us) client_negotiation.cc:240] Negotiation successful
0812 07:05:04.948914 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":342,"threads_started":1}
I20260812 07:05:04.952142 17725 heartbeater.cc:347] Connected to a master server at 127.6.61.62:42145
I20260812 07:05:04.952518 17725 heartbeater.cc:464] Registering TS with master...
I20260812 07:05:04.953316 17725 heartbeater.cc:511] Master 127.6.61.62:42145 requested a full tablet report, sending...
I20260812 07:05:04.955016 17352 ts_manager.cc:194] Registered new tserver with Master: 4bf2c6ecdd814a79ade88b4bfcd478a6 (127.6.61.2:40381)
I20260812 07:05:04.955855 17352 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.61.2:40353
W20260812 07:05:05.303570 17728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:05:05.303923 17728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:05:05.304091 17728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:05:05.311688 17728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:05:05.311851 17728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:05:05.312038 17728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.3
I20260812 07:05:05.321753 17728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:45243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:42145
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.3
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:05:05.323247 17728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:05:05.325174 17728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:05:05.340337 17735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:05.340407 17739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:05.340380 17736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:05:05.343621 17728 server_base.cc:1034] running on GCE node
I20260812 07:05:05.344388 17728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:05:05.345460 17728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:05:05.346686 17728 hybrid_clock.cc:648] HybridClock initialized: now 1786518305346580 us; error 91 us; skew 500 ppm
I20260812 07:05:05.347002 17728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 07:05:05.349431 17728 webserver.cc:460] Webserver started at http://127.6.61.3:39363/ using document root <none> and password file <none>
I20260812 07:05:05.350195 17728 fs_manager.cc:359] Metadata directory not provided
I20260812 07:05:05.350287 17728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:05:05.350587 17728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:05:05.353628 17728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/data/instance:
uuid: "e82860c781804e69ac59a4caaca565ef"
format_stamp: "Formatted at 2026-08-12 07:05:05 on dist-test-slave-69rg"
server_key: "d919c28e4a99242c0695ee0d0e2c27a5"
server_key_iv: "8f9a0a7ff037ba7e11082d4d588e1b5f"
server_key_version: "encryptionkey@0"
I20260812 07:05:05.354609 17728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance:
uuid: "e82860c781804e69ac59a4caaca565ef"
format_stamp: "Formatted at 2026-08-12 07:05:05 on dist-test-slave-69rg"
server_key: "d919c28e4a99242c0695ee0d0e2c27a5"
server_key_iv: "8f9a0a7ff037ba7e11082d4d588e1b5f"
server_key_version: "encryptionkey@0"
I20260812 07:05:05.359931 17728 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 07:05:05.363631 17746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:05.365222 17728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 07:05:05.365427 17728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/wal
uuid: "e82860c781804e69ac59a4caaca565ef"
format_stamp: "Formatted at 2026-08-12 07:05:05 on dist-test-slave-69rg"
server_key: "d919c28e4a99242c0695ee0d0e2c27a5"
server_key_iv: "8f9a0a7ff037ba7e11082d4d588e1b5f"
server_key_version: "encryptionkey@0"
I20260812 07:05:05.365677 17728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:05:05.411269 17728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:05:05.412276 17728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:05:05.412599 17728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:05:05.413709 17728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:05:05.415472 17728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:05:05.415601 17728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:05.415746 17728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:05:05.415845 17728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:05.452733 17728 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.3:33035
I20260812 07:05:05.452827 17874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.3:33035 every 8 connection(s)
I20260812 07:05:05.454174 17728 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
I20260812 07:05:05.456310 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 17728
I20260812 07:05:05.456516 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance
I20260812 07:05:05.457063 6388 external_mini_cluster.cc:1428] Setting key f333e8a460b30e062cbfc42724060d8f
I20260812 07:05:05.467469 17569 heartbeater.cc:503] Master 127.6.61.62:42145 was elected leader, sending a full tablet report...
I20260812 07:05:05.468569 17877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:05.457184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:52553 (local address 127.6.61.62:42145)
0812 07:05:05.457691 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:05.457697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:05.459380 (+ 1683us) server_negotiation.cc:407] Connection header received
0812 07:05:05.461438 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:05.461443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:05.461509 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:05.461607 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:05.463067 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:05.463876 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:05.465497 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:05.466035 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:05.466181 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:05.467486 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:05.467491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:05.467494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:05:05.467513 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:05.467555 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:05:05.467567 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:05:05.467671 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:05.468029 (+ 358us) server_negotiation.cc:299] Negotiation successful
0812 07:05:05.468144 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":242,"threads_started":1}
I20260812 07:05:05.469769 17879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:05.457942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:42145 (local address 127.6.61.3:52553)
0812 07:05:05.459223 (+ 1281us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:05.459270 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:05.461156 (+ 1886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:05.461868 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:05.461881 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:05.462002 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:05:05.462803 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:05.462830 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:05.464037 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:05.464041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:05.465343 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:05.465356 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:05.466249 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:05.466255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:05.466467 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:05.467137 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:05:05.467139 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:05:05.467178 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:05:05.467266 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:05:05.467669 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:05:05.467682 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:05:05.467693 (+ 11us) client_negotiation.cc:769] Sending connection context
0812 07:05:05.467944 (+ 251us) client_negotiation.cc:240] Negotiation successful
0812 07:05:05.468428 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":414,"threads_started":1}
I20260812 07:05:05.471410 17875 heartbeater.cc:347] Connected to a master server at 127.6.61.62:42145
I20260812 07:05:05.471771 17875 heartbeater.cc:464] Registering TS with master...
I20260812 07:05:05.472597 17875 heartbeater.cc:511] Master 127.6.61.62:42145 requested a full tablet report, sending...
I20260812 07:05:05.474088 17352 ts_manager.cc:194] Registered new tserver with Master: e82860c781804e69ac59a4caaca565ef (127.6.61.3:33035)
I20260812 07:05:05.474922 17352 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.61.3:52553
I20260812 07:05:05.485575 6388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 07:05:05.492467 17877 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:05:05.490729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35862 (local address 127.6.61.62:42145)
0812 07:05:05.490924 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:05.490933 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:05.491378 (+ 445us) server_negotiation.cc:407] Connection header received
0812 07:05:05.491581 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:05.491592 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:05.491660 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:05.491776 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:05.492238 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35862: BlockingRecv error: recv got EOF from 127.0.0.1:35862 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 07:05:05.493291 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 17421
I20260812 07:05:05.501754 17886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:05.494394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36456 (local address 127.6.61.1:43327)
0812 07:05:05.495038 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:05.495045 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:05.495073 (+ 28us) server_negotiation.cc:407] Connection header received
0812 07:05:05.495137 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:05.495144 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:05.495361 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:05.495500 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:05.495903 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:05.496943 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:05.498393 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:05.499016 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:05.499125 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:05.500293 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:05.500301 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:05.500316 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:05:05.500899 (+ 583us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:05.500950 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:05:05.500975 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:05:05.501068 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:05.501205 (+ 137us) server_negotiation.cc:299] Negotiation successful
0812 07:05:05.501372 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":339,"threads_started":1}
I20260812 07:05:05.503295 17564 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:05:05.585502 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 17421
I20260812 07:05:05.606269 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 17573
I20260812 07:05:05.614423 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:05.607313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59544 (local address 127.6.61.2:40381)
0812 07:05:05.607932 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:05.607939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:05.607967 (+ 28us) server_negotiation.cc:407] Connection header received
0812 07:05:05.608031 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:05.608038 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:05.608263 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:05.608394 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:05.608751 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:05.609746 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:05.611056 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:05.611650 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:05.611756 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:05.612813 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:05.612822 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:05.612836 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:05:05.613460 (+ 624us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:05.613509 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:05:05.613529 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:05:05.613631 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:05.613882 (+ 251us) server_negotiation.cc:299] Negotiation successful
0812 07:05:05.614012 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":336,"threads_started":1}
I20260812 07:05:05.615921 17720 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:05:05.703078 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 17573
I20260812 07:05:05.723676 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 17728
I20260812 07:05:05.732336 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:05.724709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34590 (local address 127.6.61.3:33035)
0812 07:05:05.725251 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:05.725258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:05.725290 (+ 32us) server_negotiation.cc:407] Connection header received
0812 07:05:05.725354 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:05.725361 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:05.725584 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:05.725760 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:05.726302 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:05.727356 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:05.728661 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:05.729239 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:05.729385 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:05.730744 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:05.730754 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:05.730770 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:05:05.731434 (+ 664us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:05.731487 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:05:05.731510 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:05:05.731647 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:05.731816 (+ 169us) server_negotiation.cc:299] Negotiation successful
0812 07:05:05.731965 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":231,"threads_started":1}
I20260812 07:05:05.733897 17870 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:05:05.816929 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 17728
I20260812 07:05:05.840857 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 17315
I20260812 07:05:05.842371 17383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:05:05.948215 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 17315
2026-08-12T07:05:05Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2639 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-12T07:05:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:05:06Z Disabled control of system clock
I20260812 07:05:06.038275 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:34517
--webserver_interface=127.6.61.62
--webserver_port=0
--builtin_ntp_servers=127.6.61.20:44255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:34517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 07:05:06.390595 17899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:05:06.390903 17899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:05:06.390990 17899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:05:06.398451 17899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:05:06.398564 17899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:05:06.398602 17899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:05:06.398653 17899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:05:06.398698 17899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:05:06.408090 17899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:44255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:34517
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:34517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:05:06.409368 17899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:05:06.411137 17899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:05:06.423374 17905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:06.423429 17906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:06.423483 17908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:05:06.424401 17899 server_base.cc:1034] running on GCE node
I20260812 07:05:06.425374 17899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:05:06.426945 17899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:05:06.428200 17899 hybrid_clock.cc:648] HybridClock initialized: now 1786518306428168 us; error 56 us; skew 500 ppm
I20260812 07:05:06.428610 17899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 07:05:06.431062 17899 webserver.cc:460] Webserver started at http://127.6.61.62:34581/ using document root <none> and password file <none>
I20260812 07:05:06.431794 17899 fs_manager.cc:359] Metadata directory not provided
I20260812 07:05:06.431933 17899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:05:06.432282 17899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:05:06.435372 17899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "a72b4464bb134b01822b7d924924ca59"
format_stamp: "Formatted at 2026-08-12 07:05:06 on dist-test-slave-69rg"
server_key: "b7d8d671dc42025eb1722b08332e2381"
server_key_iv: "1b11fee0df528256f69af6b3ea73547a"
server_key_version: "encryptionkey@0"
I20260812 07:05:06.436278 17899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "a72b4464bb134b01822b7d924924ca59"
format_stamp: "Formatted at 2026-08-12 07:05:06 on dist-test-slave-69rg"
server_key: "b7d8d671dc42025eb1722b08332e2381"
server_key_iv: "1b11fee0df528256f69af6b3ea73547a"
server_key_version: "encryptionkey@0"
I20260812 07:05:06.441483 17899 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 07:05:06.445096 17914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:06.446571 17899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:05:06.446759 17899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "a72b4464bb134b01822b7d924924ca59"
format_stamp: "Formatted at 2026-08-12 07:05:06 on dist-test-slave-69rg"
server_key: "b7d8d671dc42025eb1722b08332e2381"
server_key_iv: "1b11fee0df528256f69af6b3ea73547a"
server_key_version: "encryptionkey@0"
I20260812 07:05:06.446985 17899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:05:06.452852 17899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:05:06.453619 17899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:05:06.454002 17899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:05:06.473230 17899 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:34517
I20260812 07:05:06.473279 17976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:34517 every 8 connection(s)
I20260812 07:05:06.474648 17899 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:05:06.477177 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 17899
I20260812 07:05:06.477420 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:05:06.478194 6388 external_mini_cluster.cc:1428] Setting key 9df2fc5bf66828749b580122190409ab
I20260812 07:05:06.478704 17977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:05:06.488145 17981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:06.480426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40372 (local address 127.6.61.62:34517)
0812 07:05:06.481543 (+ 1117us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:06.481560 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:06.481610 (+ 50us) server_negotiation.cc:407] Connection header received
0812 07:05:06.483835 (+ 2225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:06.483865 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:06.484469 (+ 604us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:06.484925 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:06.486032 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:06.486041 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:06.486060 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:05:06.486077 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:06.486178 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:05:06.486204 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:05:06.486434 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:06.486586 (+ 152us) server_negotiation.cc:299] Negotiation successful
0812 07:05:06.486897 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":336,"threads_started":1}
I20260812 07:05:06.493508 17977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a72b4464bb134b01822b7d924924ca59: Bootstrap starting.
I20260812 07:05:06.496688 17977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a72b4464bb134b01822b7d924924ca59: Neither blocks nor log segments found. Creating new log.
I20260812 07:05:06.497800 17977 log.cc:824] T 00000000000000000000000000000000 P a72b4464bb134b01822b7d924924ca59: Log is configured to *not* fsync() on all Append() calls
I20260812 07:05:06.501233 17977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a72b4464bb134b01822b7d924924ca59: No bootstrap required, opened a new log
I20260812 07:05:06.508713 17977 raft_consensus.cc:348] T 00000000000000000000000000000000 P a72b4464bb134b01822b7d924924ca59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a72b4464bb134b01822b7d924924ca59" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34517 } }
I20260812 07:05:06.509124 17977 raft_consensus.cc:374] T 00000000000000000000000000000000 P a72b4464bb134b01822b7d924924ca59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:05:06.509243 17977 raft_consensus.cc:729] T 00000000000000000000000000000000 P a72b4464bb134b01822b7d924924ca59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72b4464bb134b01822b7d924924ca59, State: Initialized, Role: FOLLOWER
I20260812 07:05:06.509872 17977 consensus_queue.cc:260] T 00000000000000000000000000000000 P a72b4464bb134b01822b7d924924ca59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a72b4464bb134b01822b7d924924ca59" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34517 } }
I20260812 07:05:06.510145 17977 raft_consensus.cc:388] T 00000000000000000000000000000000 P a72b4464bb134b01822b7d924924ca59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:05:06.510293 17977 raft_consensus.cc:482] T 00000000000000000000000000000000 P a72b4464bb134b01822b7d924924ca59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:05:06.510478 17977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a72b4464bb134b01822b7d924924ca59 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:05:06.512791 17977 raft_consensus.cc:504] T 00000000000000000000000000000000 P a72b4464bb134b01822b7d924924ca59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a72b4464bb134b01822b7d924924ca59" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34517 } }
I20260812 07:05:06.513311 17977 leader_election.cc:302] T 00000000000000000000000000000000 P a72b4464bb134b01822b7d924924ca59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a72b4464bb134b01822b7d924924ca59; no voters:
I20260812 07:05:06.514037 17977 leader_election.cc:288] T 00000000000000000000000000000000 P a72b4464bb134b01822b7d924924ca59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:05:06.514191 17983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a72b4464bb134b01822b7d924924ca59 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:05:06.514931 17983 raft_consensus.cc:686] T 00000000000000000000000000000000 P a72b4464bb134b01822b7d924924ca59 [term 1 LEADER]: Becoming Leader. State: Replica: a72b4464bb134b01822b7d924924ca59, State: Running, Role: LEADER
I20260812 07:05:06.515465 17983 consensus_queue.cc:237] T 00000000000000000000000000000000 P a72b4464bb134b01822b7d924924ca59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a72b4464bb134b01822b7d924924ca59" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34517 } }
I20260812 07:05:06.516289 17977 sys_catalog.cc:564] T 00000000000000000000000000000000 P a72b4464bb134b01822b7d924924ca59 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:05:06.518373 17984 sys_catalog.cc:455] T 00000000000000000000000000000000 P a72b4464bb134b01822b7d924924ca59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a72b4464bb134b01822b7d924924ca59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a72b4464bb134b01822b7d924924ca59" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34517 } } }
I20260812 07:05:06.518854 17984 sys_catalog.cc:458] T 00000000000000000000000000000000 P a72b4464bb134b01822b7d924924ca59 [sys.catalog]: This master's current role is: LEADER
I20260812 07:05:06.519810 17988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:05:06.519719 17985 sys_catalog.cc:455] T 00000000000000000000000000000000 P a72b4464bb134b01822b7d924924ca59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a72b4464bb134b01822b7d924924ca59. Latest consensus state: current_term: 1 leader_uuid: "a72b4464bb134b01822b7d924924ca59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a72b4464bb134b01822b7d924924ca59" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34517 } } }
I20260812 07:05:06.520872 17985 sys_catalog.cc:458] T 00000000000000000000000000000000 P a72b4464bb134b01822b7d924924ca59 [sys.catalog]: This master's current role is: LEADER
I20260812 07:05:06.525663 17988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:05:06.537213 17988 catalog_manager.cc:1347] Generated new cluster ID: 2d9c0fa999ff4dfc9cbe5a810437d830
I20260812 07:05:06.537341 17988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:05:06.552737 17988 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:05:06.553220 17988 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:05:06.564646 17988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a72b4464bb134b01822b7d924924ca59: Generated new TSK 0
I20260812 07:05:06.565513 17988 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:05:06.570537 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.1:0
--local_ip_for_outbound_sockets=127.6.61.1
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:34517
--builtin_ntp_servers=127.6.61.20:44255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 07:05:06.950243 18008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:05:06.950603 18008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:05:06.950778 18008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:05:06.958539 18008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:05:06.958647 18008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:05:06.958844 18008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.1
I20260812 07:05:06.968322 18008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:44255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:34517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.1
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:05:06.969678 18008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:05:06.971554 18008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:05:06.985033 18013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:06.985090 18014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:06.985635 18016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:05:06.986456 18008 server_base.cc:1034] running on GCE node
I20260812 07:05:06.987130 18008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:05:06.988118 18008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:05:06.989364 18008 hybrid_clock.cc:648] HybridClock initialized: now 1786518306989316 us; error 60 us; skew 500 ppm
I20260812 07:05:06.989704 18008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 07:05:06.992292 18008 webserver.cc:460] Webserver started at http://127.6.61.1:39661/ using document root <none> and password file <none>
I20260812 07:05:06.993013 18008 fs_manager.cc:359] Metadata directory not provided
I20260812 07:05:06.993144 18008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:05:06.993522 18008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:05:06.996513 18008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/data/instance:
uuid: "5193d903179b4b97a5debc534dfe22f9"
format_stamp: "Formatted at 2026-08-12 07:05:06 on dist-test-slave-69rg"
server_key: "39f26cf35fe8513e6b981ebcb94341aa"
server_key_iv: "c38acfde5aba398b1a008235c92db3ca"
server_key_version: "encryptionkey@0"
I20260812 07:05:06.997330 18008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance:
uuid: "5193d903179b4b97a5debc534dfe22f9"
format_stamp: "Formatted at 2026-08-12 07:05:06 on dist-test-slave-69rg"
server_key: "39f26cf35fe8513e6b981ebcb94341aa"
server_key_iv: "c38acfde5aba398b1a008235c92db3ca"
server_key_version: "encryptionkey@0"
I20260812 07:05:07.003321 18008 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260812 07:05:07.007376 18024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:07.008987 18008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:05:07.009222 18008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/wal
uuid: "5193d903179b4b97a5debc534dfe22f9"
format_stamp: "Formatted at 2026-08-12 07:05:06 on dist-test-slave-69rg"
server_key: "39f26cf35fe8513e6b981ebcb94341aa"
server_key_iv: "c38acfde5aba398b1a008235c92db3ca"
server_key_version: "encryptionkey@0"
I20260812 07:05:07.009495 18008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:05:07.016212 18008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:05:07.017237 18008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:05:07.017560 18008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:05:07.018821 18008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:05:07.020401 18008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:05:07.020542 18008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:07.020699 18008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:05:07.020798 18008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:07.057988 18008 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.1:44321
I20260812 07:05:07.058099 18152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.1:44321 every 8 connection(s)
I20260812 07:05:07.059505 18008 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
I20260812 07:05:07.065732 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 18008
I20260812 07:05:07.066090 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance
I20260812 07:05:07.066833 6388 external_mini_cluster.cc:1428] Setting key 13d846d975c27b1441b2349693696b80
I20260812 07:05:07.069015 18155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:07.062357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:53003 (local address 127.6.61.62:34517)
0812 07:05:07.062862 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:07.062867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:07.064044 (+ 1177us) server_negotiation.cc:407] Connection header received
0812 07:05:07.066146 (+ 2102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:07.066153 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:07.066250 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:07.066390 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:07.068205 (+ 1815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:07.068210 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:07.068213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:05:07.068232 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:07.068278 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:05:07.068291 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:05:07.068448 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:07.068655 (+ 207us) server_negotiation.cc:299] Negotiation successful
0812 07:05:07.068728 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":272,"threads_started":1}
I20260812 07:05:07.070073 18156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:07.062821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34517 (local address 127.6.61.1:53003)
0812 07:05:07.063849 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:07.063890 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:07.065722 (+ 1832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:07.066664 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:07.066681 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:07.066835 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:05:07.067791 (+ 956us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:05:07.067794 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:05:07.067851 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:05:07.067975 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:05:07.068408 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:05:07.068419 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:05:07.068431 (+ 12us) client_negotiation.cc:769] Sending connection context
0812 07:05:07.068739 (+ 308us) client_negotiation.cc:240] Negotiation successful
0812 07:05:07.069017 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":320,"threads_started":1}
I20260812 07:05:07.070237 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.2:0
--local_ip_for_outbound_sockets=127.6.61.2
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:34517
--builtin_ntp_servers=127.6.61.20:44255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260812 07:05:07.071663 18153 heartbeater.cc:347] Connected to a master server at 127.6.61.62:34517
I20260812 07:05:07.071931 18153 heartbeater.cc:464] Registering TS with master...
I20260812 07:05:07.072515 18153 heartbeater.cc:511] Master 127.6.61.62:34517 requested a full tablet report, sending...
I20260812 07:05:07.074669 17933 ts_manager.cc:194] Registered new tserver with Master: 5193d903179b4b97a5debc534dfe22f9 (127.6.61.1:44321)
W20260812 07:05:07.421391 18157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:05:07.421727 18157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:05:07.421890 18157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:05:07.429195 18157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:05:07.429328 18157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:05:07.429509 18157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.2
I20260812 07:05:07.439334 18157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:44255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:34517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.2
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:05:07.440747 18157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:05:07.442749 18157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:05:07.456342 18166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:07.456408 18163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:07.456431 18164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:05:07.457427 18157 server_base.cc:1034] running on GCE node
I20260812 07:05:07.458199 18157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:05:07.459106 18157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:05:07.460357 18157 hybrid_clock.cc:648] HybridClock initialized: now 1786518307460304 us; error 48 us; skew 500 ppm
I20260812 07:05:07.460687 18157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 07:05:07.467681 18157 webserver.cc:460] Webserver started at http://127.6.61.2:41687/ using document root <none> and password file <none>
I20260812 07:05:07.468433 18157 fs_manager.cc:359] Metadata directory not provided
I20260812 07:05:07.468577 18157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:05:07.468933 18157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:05:07.472074 18157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/data/instance:
uuid: "663df8ec26f74934b5f27bb5475b29fd"
format_stamp: "Formatted at 2026-08-12 07:05:07 on dist-test-slave-69rg"
server_key: "5ffd8fc30f70fbefbdaa592ef0aece8d"
server_key_iv: "e859e3763eda160b575aaf5a5144b439"
server_key_version: "encryptionkey@0"
I20260812 07:05:07.473002 18157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance:
uuid: "663df8ec26f74934b5f27bb5475b29fd"
format_stamp: "Formatted at 2026-08-12 07:05:07 on dist-test-slave-69rg"
server_key: "5ffd8fc30f70fbefbdaa592ef0aece8d"
server_key_iv: "e859e3763eda160b575aaf5a5144b439"
server_key_version: "encryptionkey@0"
I20260812 07:05:07.478340 18157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260812 07:05:07.481986 18174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:07.483405 18157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:05:07.483588 18157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/wal
uuid: "663df8ec26f74934b5f27bb5475b29fd"
format_stamp: "Formatted at 2026-08-12 07:05:07 on dist-test-slave-69rg"
server_key: "5ffd8fc30f70fbefbdaa592ef0aece8d"
server_key_iv: "e859e3763eda160b575aaf5a5144b439"
server_key_version: "encryptionkey@0"
I20260812 07:05:07.483824 18157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:05:07.489960 18157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:05:07.490854 18157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:05:07.491168 18157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:05:07.492153 18157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:05:07.493718 18157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:05:07.493865 18157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:07.494010 18157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:05:07.494102 18157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:07.530447 18157 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.2:40183
I20260812 07:05:07.530558 18304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.2:40183 every 8 connection(s)
I20260812 07:05:07.531908 18157 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
I20260812 07:05:07.540458 18155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:07.534638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:42833 (local address 127.6.61.62:34517)
0812 07:05:07.534837 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:07.534842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:07.536239 (+ 1397us) server_negotiation.cc:407] Connection header received
0812 07:05:07.538007 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:07.538013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:07.538103 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:07.538222 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:07.539633 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:07.539639 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:07.539643 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:05:07.539665 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:07.539707 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:05:07.539719 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:05:07.539879 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:07.540079 (+ 200us) server_negotiation.cc:299] Negotiation successful
0812 07:05:07.540157 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 07:05:07.541425 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:07.535158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34517 (local address 127.6.61.2:42833)
0812 07:05:07.536068 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:07.536113 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:07.537699 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:07.538467 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:07.538476 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:07.538564 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:05:07.539302 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:05:07.539305 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:05:07.539352 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:05:07.539439 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:05:07.539825 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:05:07.539837 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:05:07.539847 (+ 10us) client_negotiation.cc:769] Sending connection context
0812 07:05:07.540149 (+ 302us) client_negotiation.cc:240] Negotiation successful
0812 07:05:07.540426 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":275,"threads_started":1}
I20260812 07:05:07.541864 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 18157
I20260812 07:05:07.542104 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance
I20260812 07:05:07.542569 6388 external_mini_cluster.cc:1428] Setting key 75d7a5e9255ad1c597807304da84e4a7
I20260812 07:05:07.543025 18305 heartbeater.cc:347] Connected to a master server at 127.6.61.62:34517
I20260812 07:05:07.543346 18305 heartbeater.cc:464] Registering TS with master...
I20260812 07:05:07.543963 18305 heartbeater.cc:511] Master 127.6.61.62:34517 requested a full tablet report, sending...
I20260812 07:05:07.545404 17933 ts_manager.cc:194] Registered new tserver with Master: 663df8ec26f74934b5f27bb5475b29fd (127.6.61.2:40183)
I20260812 07:05:07.545461 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.3:0
--local_ip_for_outbound_sockets=127.6.61.3
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:34517
--builtin_ntp_servers=127.6.61.20:44255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 07:05:07.892957 18310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:05:07.893352 18310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:05:07.893500 18310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:05:07.900870 18310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:05:07.900988 18310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:05:07.901099 18310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.3
I20260812 07:05:07.910259 18310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:44255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:34517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.3
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:05:07.911525 18310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:05:07.913303 18310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:05:07.930197 18317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:07.930272 18320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:07.930233 18318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:05:07.931105 18310 server_base.cc:1034] running on GCE node
I20260812 07:05:07.931845 18310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:05:07.932910 18310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:05:07.934180 18310 hybrid_clock.cc:648] HybridClock initialized: now 1786518307934123 us; error 45 us; skew 500 ppm
I20260812 07:05:07.934508 18310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 07:05:07.937016 18310 webserver.cc:460] Webserver started at http://127.6.61.3:46737/ using document root <none> and password file <none>
I20260812 07:05:07.937740 18310 fs_manager.cc:359] Metadata directory not provided
I20260812 07:05:07.937907 18310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:05:07.938266 18310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:05:07.941306 18310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/data/instance:
uuid: "eaf5dd6f04274361acbf732534b7b709"
format_stamp: "Formatted at 2026-08-12 07:05:07 on dist-test-slave-69rg"
server_key: "01dfc9dee19ee0a1648af5b32ad9f31a"
server_key_iv: "e47e19b49747746f1953d749b5dd2900"
server_key_version: "encryptionkey@0"
I20260812 07:05:07.942363 18310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance:
uuid: "eaf5dd6f04274361acbf732534b7b709"
format_stamp: "Formatted at 2026-08-12 07:05:07 on dist-test-slave-69rg"
server_key: "01dfc9dee19ee0a1648af5b32ad9f31a"
server_key_iv: "e47e19b49747746f1953d749b5dd2900"
server_key_version: "encryptionkey@0"
I20260812 07:05:07.948944 18310 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 07:05:07.952798 18329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:07.954272 18310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:05:07.954417 18310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/wal
uuid: "eaf5dd6f04274361acbf732534b7b709"
format_stamp: "Formatted at 2026-08-12 07:05:07 on dist-test-slave-69rg"
server_key: "01dfc9dee19ee0a1648af5b32ad9f31a"
server_key_iv: "e47e19b49747746f1953d749b5dd2900"
server_key_version: "encryptionkey@0"
I20260812 07:05:07.954617 18310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:05:07.960827 18310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:05:07.961615 18310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:05:07.961920 18310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:05:07.962939 18310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:05:07.964491 18310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:05:07.964586 18310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:07.964668 18310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:05:07.964725 18310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:08.001034 18310 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.3:34265
I20260812 07:05:08.001092 18462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.3:34265 every 8 connection(s)
I20260812 07:05:08.002890 18310 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
I20260812 07:05:08.007423 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 18310
I20260812 07:05:08.007648 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance
I20260812 07:05:08.008088 6388 external_mini_cluster.cc:1428] Setting key 2bf5e3f4cbb4ca8b4ea0df9900f3d930
I20260812 07:05:08.011874 18155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:08.005535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:39167 (local address 127.6.61.62:34517)
0812 07:05:08.005759 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:08.005766 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:08.007263 (+ 1497us) server_negotiation.cc:407] Connection header received
0812 07:05:08.009207 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:08.009213 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:08.009294 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:08.009409 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:08.010914 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:08.010919 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:08.010922 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:05:08.010938 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:08.011008 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:05:08.011028 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:05:08.011154 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:08.011541 (+ 387us) server_negotiation.cc:299] Negotiation successful
0812 07:05:08.011617 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 07:05:08.012619 18466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:08.006160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:34517 (local address 127.6.61.3:39167)
0812 07:05:08.007081 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:08.007127 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:08.008907 (+ 1780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:08.009652 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:08.009660 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:08.009751 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:05:08.010494 (+ 743us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:05:08.010500 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:05:08.010556 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:05:08.010705 (+ 149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:05:08.011174 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:05:08.011186 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:05:08.011194 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 07:05:08.011474 (+ 280us) client_negotiation.cc:240] Negotiation successful
0812 07:05:08.011679 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":291,"threads_started":1}
I20260812 07:05:08.014232 18463 heartbeater.cc:347] Connected to a master server at 127.6.61.62:34517
I20260812 07:05:08.014547 18463 heartbeater.cc:464] Registering TS with master...
I20260812 07:05:08.015169 18463 heartbeater.cc:511] Master 127.6.61.62:34517 requested a full tablet report, sending...
I20260812 07:05:08.016633 17933 ts_manager.cc:194] Registered new tserver with Master: eaf5dd6f04274361acbf732534b7b709 (127.6.61.3:34265)
I20260812 07:05:08.023900 6388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 07:05:08.030306 18155 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:05:08.028632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40378 (local address 127.6.61.62:34517)
0812 07:05:08.028866 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:08.028876 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:08.029327 (+ 451us) server_negotiation.cc:407] Connection header received
0812 07:05:08.029458 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:08.029462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:08.029514 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:08.029594 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:08.030000 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40378: BlockingRecv error: recv got EOF from 127.0.0.1:40378 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 07:05:08.030622 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 18008
I20260812 07:05:08.034852 18474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:08.032174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59388 (local address 127.6.61.1:44321)
0812 07:05:08.032742 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:08.032749 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:08.032775 (+ 26us) server_negotiation.cc:407] Connection header received
0812 07:05:08.032834 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:08.032840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:08.033038 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:08.033178 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:08.034114 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:08.034120 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:08.034132 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:05:08.034151 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:08.034192 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:05:08.034206 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:05:08.034313 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:08.034401 (+ 88us) server_negotiation.cc:299] Negotiation successful
0812 07:05:08.034516 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":279,"threads_started":1}
I20260812 07:05:08.036099 18147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:05:08.122116 18153 heartbeater.cc:503] Master 127.6.61.62:34517 was elected leader, sending a full tablet report...
I20260812 07:05:08.123494 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 18008
W20260812 07:05:08.143831 17917 connection.cc:535] server connection from 127.6.61.1:53003 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 07:05:08.144009 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 18157
I20260812 07:05:08.148173 18477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:08.145077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53374 (local address 127.6.61.2:40183)
0812 07:05:08.145722 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:08.145730 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:08.145760 (+ 30us) server_negotiation.cc:407] Connection header received
0812 07:05:08.145874 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:08.145882 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:08.146122 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:08.146283 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:08.147496 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:08.147504 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:08.147517 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:05:08.147531 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:08.147581 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:05:08.147596 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:05:08.147735 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:08.147835 (+ 100us) server_negotiation.cc:299] Negotiation successful
0812 07:05:08.147929 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":311,"threads_started":1}
I20260812 07:05:08.149509 18300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:05:08.231580 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 18157
I20260812 07:05:08.253194 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 18310
I20260812 07:05:08.257762 18479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:08.254444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60858 (local address 127.6.61.3:34265)
0812 07:05:08.255163 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:08.255171 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:08.255205 (+ 34us) server_negotiation.cc:407] Connection header received
0812 07:05:08.255273 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:08.255280 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:08.255509 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:08.255698 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:08.256958 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:08.256965 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:08.256976 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:05:08.256993 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:08.257033 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:05:08.257047 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:05:08.257251 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:08.257340 (+ 89us) server_negotiation.cc:299] Negotiation successful
0812 07:05:08.257448 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":341,"threads_started":1}
I20260812 07:05:08.259225 18454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:05:08.342808 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 18310
I20260812 07:05:08.364836 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 17899
I20260812 07:05:08.366621 17969 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:05:08.471352 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 17899
2026-08-12T07:05:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2524 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:05:08 dist-test-slave-69rg krb5kdc[18489](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:05:08 dist-test-slave-69rg krb5kdc[18489](info): set up 2 sockets
Aug 12 07:05:08 dist-test-slave-69rg krb5kdc[18489](info): commencing operation
krb5kdc: starting...
W20260812 07:05:09.703728 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.185s user 0.001s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:05:09 dist-test-slave-69rg krb5kdc[18489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:05:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:05:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
I20260812 07:05:10.043165 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:44841
--webserver_interface=127.6.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:42463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:44841
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:05:10.398866 18503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:05:10.399435 18503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:05:10.399622 18503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:05:10.408365 18503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:05:10.408494 18503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:05:10.408593 18503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:05:10.408702 18503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:05:10.408792 18503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:05:10.421284 18503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:42463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:44841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:44841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:05:10.422890 18503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:05:10.424404 18503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:05:10.437708 18512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:10.437732 18510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:10.438107 18509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:05:10.438665 18503 server_base.cc:1034] running on GCE node
I20260812 07:05:10.439564 18503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:05:10.441177 18503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:05:10.442426 18503 hybrid_clock.cc:648] HybridClock initialized: now 1786518310442362 us; error 53 us; skew 500 ppm
I20260812 07:05:10.442816 18503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:05:10 dist-test-slave-69rg krb5kdc[18489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518310, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:05:10.446473 18503 init.cc:377] Logged in from keytab as kudu/127.6.61.62@KRBTEST.COM (short username kudu)
I20260812 07:05:10.448359 18503 webserver.cc:460] Webserver started at http://127.6.61.62:39371/ using document root <none> and password file <none>
I20260812 07:05:10.449107 18503 fs_manager.cc:359] Metadata directory not provided
I20260812 07:05:10.449234 18503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:05:10.449527 18503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:05:10.453078 18503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "d17d48231e9443c89144db3c3a73b40b"
format_stamp: "Formatted at 2026-08-12 07:05:10 on dist-test-slave-69rg"
server_key: "5560c7e2b4d31ba028312cce1e18af88"
server_key_iv: "247d732c514ac0022675ec9143a024a3"
server_key_version: "encryptionkey@0"
I20260812 07:05:10.454022 18503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "d17d48231e9443c89144db3c3a73b40b"
format_stamp: "Formatted at 2026-08-12 07:05:10 on dist-test-slave-69rg"
server_key: "5560c7e2b4d31ba028312cce1e18af88"
server_key_iv: "247d732c514ac0022675ec9143a024a3"
server_key_version: "encryptionkey@0"
I20260812 07:05:10.459924 18503 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 07:05:10.463913 18519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:10.465523 18503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 07:05:10.465703 18503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "d17d48231e9443c89144db3c3a73b40b"
format_stamp: "Formatted at 2026-08-12 07:05:10 on dist-test-slave-69rg"
server_key: "5560c7e2b4d31ba028312cce1e18af88"
server_key_iv: "247d732c514ac0022675ec9143a024a3"
server_key_version: "encryptionkey@0"
I20260812 07:05:10.465937 18503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:05:10.490351 18503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:05:10.494939 18503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:05:10.495186 18503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:05:10.515190 18503 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:44841
I20260812 07:05:10.515280 18580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:44841 every 8 connection(s)
I20260812 07:05:10.516655 18503 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:05:10.517839 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 18503
I20260812 07:05:10.518122 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:05:10.518819 6388 external_mini_cluster.cc:1428] Setting key 7f4aedc89ef9318a021b06e4343285a2
I20260812 07:05:10.521310 18581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 07:05:10 dist-test-slave-69rg krb5kdc[18489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:05:10.535672 18581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d17d48231e9443c89144db3c3a73b40b: Bootstrap starting.
I20260812 07:05:10.540105 18581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d17d48231e9443c89144db3c3a73b40b: Neither blocks nor log segments found. Creating new log.
I20260812 07:05:10.541457 18581 log.cc:824] T 00000000000000000000000000000000 P d17d48231e9443c89144db3c3a73b40b: Log is configured to *not* fsync() on all Append() calls
I20260812 07:05:10.544988 18581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d17d48231e9443c89144db3c3a73b40b: No bootstrap required, opened a new log
I20260812 07:05:10.545492 18586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:10.520909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42860 (local address 127.6.61.62:44841)
0812 07:05:10.521807 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:10.521848 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:10.521906 (+ 58us) server_negotiation.cc:407] Connection header received
0812 07:05:10.524326 (+ 2420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:10.524359 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:10.524978 (+ 619us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:10.525501 (+ 523us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:10.527254 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:10.529411 (+ 2157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:10.530961 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:10.532087 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:10.532305 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:10.536673 (+ 4368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:10.536685 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:10.536706 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:05:10.536754 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:10.540919 (+ 4165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:10.541848 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:10.541857 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:10.541867 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:10.542074 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:10.542575 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:10.542579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:10.542581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:10.543209 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:05:10.543536 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:10.544022 (+ 486us) server_negotiation.cc:299] Negotiation successful
0812 07:05:10.544445 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":442,"threads_started":1}
I20260812 07:05:10.553463 18581 raft_consensus.cc:348] T 00000000000000000000000000000000 P d17d48231e9443c89144db3c3a73b40b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17d48231e9443c89144db3c3a73b40b" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 44841 } }
I20260812 07:05:10.553917 18581 raft_consensus.cc:374] T 00000000000000000000000000000000 P d17d48231e9443c89144db3c3a73b40b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:05:10.554033 18581 raft_consensus.cc:729] T 00000000000000000000000000000000 P d17d48231e9443c89144db3c3a73b40b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d17d48231e9443c89144db3c3a73b40b, State: Initialized, Role: FOLLOWER
I20260812 07:05:10.554750 18581 consensus_queue.cc:260] T 00000000000000000000000000000000 P d17d48231e9443c89144db3c3a73b40b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17d48231e9443c89144db3c3a73b40b" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 44841 } }
I20260812 07:05:10.555051 18581 raft_consensus.cc:388] T 00000000000000000000000000000000 P d17d48231e9443c89144db3c3a73b40b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:05:10.555245 18581 raft_consensus.cc:482] T 00000000000000000000000000000000 P d17d48231e9443c89144db3c3a73b40b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:05:10.555455 18581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d17d48231e9443c89144db3c3a73b40b [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:05:10.557648 18581 raft_consensus.cc:504] T 00000000000000000000000000000000 P d17d48231e9443c89144db3c3a73b40b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17d48231e9443c89144db3c3a73b40b" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 44841 } }
I20260812 07:05:10.558254 18581 leader_election.cc:302] T 00000000000000000000000000000000 P d17d48231e9443c89144db3c3a73b40b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d17d48231e9443c89144db3c3a73b40b; no voters:
I20260812 07:05:10.558852 18581 leader_election.cc:288] T 00000000000000000000000000000000 P d17d48231e9443c89144db3c3a73b40b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:05:10.559015 18588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d17d48231e9443c89144db3c3a73b40b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:05:10.560457 18588 raft_consensus.cc:686] T 00000000000000000000000000000000 P d17d48231e9443c89144db3c3a73b40b [term 1 LEADER]: Becoming Leader. State: Replica: d17d48231e9443c89144db3c3a73b40b, State: Running, Role: LEADER
I20260812 07:05:10.561026 18588 consensus_queue.cc:237] T 00000000000000000000000000000000 P d17d48231e9443c89144db3c3a73b40b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17d48231e9443c89144db3c3a73b40b" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 44841 } }
I20260812 07:05:10.561206 18581 sys_catalog.cc:564] T 00000000000000000000000000000000 P d17d48231e9443c89144db3c3a73b40b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:05:10.564016 18589 sys_catalog.cc:455] T 00000000000000000000000000000000 P d17d48231e9443c89144db3c3a73b40b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d17d48231e9443c89144db3c3a73b40b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17d48231e9443c89144db3c3a73b40b" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 44841 } } }
I20260812 07:05:10.563989 18590 sys_catalog.cc:455] T 00000000000000000000000000000000 P d17d48231e9443c89144db3c3a73b40b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d17d48231e9443c89144db3c3a73b40b. Latest consensus state: current_term: 1 leader_uuid: "d17d48231e9443c89144db3c3a73b40b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17d48231e9443c89144db3c3a73b40b" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 44841 } } }
I20260812 07:05:10.564555 18589 sys_catalog.cc:458] T 00000000000000000000000000000000 P d17d48231e9443c89144db3c3a73b40b [sys.catalog]: This master's current role is: LEADER
I20260812 07:05:10.564589 18590 sys_catalog.cc:458] T 00000000000000000000000000000000 P d17d48231e9443c89144db3c3a73b40b [sys.catalog]: This master's current role is: LEADER
I20260812 07:05:10.566195 18596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:05:10.571542 18596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:05:10.581200 18596 catalog_manager.cc:1347] Generated new cluster ID: 0ea54d9445b04368972b64713a903856
I20260812 07:05:10.581331 18596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:05:10.589419 18596 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:05:10.591339 18596 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:05:10.606066 18596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d17d48231e9443c89144db3c3a73b40b: Generated new TSK 0
I20260812 07:05:10.607239 18596 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
I20260812 07:05:10.697659 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.1:0
--local_ip_for_outbound_sockets=127.6.61.1
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:44841
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:42463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:05:11.052022 18614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:05:11.052328 18614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:05:11.052425 18614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:05:11.059813 18614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:05:11.059937 18614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:05:11.060089 18614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.1
I20260812 07:05:11.070214 18614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:42463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:44841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.1
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:05:11.071642 18614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:05:11.072880 18614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:05:11.086568 18622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:11.086634 18619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:11.086434 18620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:05:11.087774 18614 server_base.cc:1034] running on GCE node
I20260812 07:05:11.088397 18614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:05:11.089305 18614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:05:11.090502 18614 hybrid_clock.cc:648] HybridClock initialized: now 1786518311090454 us; error 41 us; skew 500 ppm
I20260812 07:05:11.090808 18614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:05:11 dist-test-slave-69rg krb5kdc[18489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518311, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:05:11.093963 18614 init.cc:377] Logged in from keytab as kudu/127.6.61.1@KRBTEST.COM (short username kudu)
I20260812 07:05:11.095477 18614 webserver.cc:460] Webserver started at http://127.6.61.1:41131/ using document root <none> and password file <none>
I20260812 07:05:11.096171 18614 fs_manager.cc:359] Metadata directory not provided
I20260812 07:05:11.096287 18614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:05:11.096524 18614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:05:11.099562 18614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/data/instance:
uuid: "cf71694cf8eb4a8d8a8a7a556d2d5967"
format_stamp: "Formatted at 2026-08-12 07:05:11 on dist-test-slave-69rg"
server_key: "f98144e2681c65fd598d85e33d8065e6"
server_key_iv: "98ecbbd26c23f43e6cb9912fe21ab8a6"
server_key_version: "encryptionkey@0"
I20260812 07:05:11.100317 18614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance:
uuid: "cf71694cf8eb4a8d8a8a7a556d2d5967"
format_stamp: "Formatted at 2026-08-12 07:05:11 on dist-test-slave-69rg"
server_key: "f98144e2681c65fd598d85e33d8065e6"
server_key_iv: "98ecbbd26c23f43e6cb9912fe21ab8a6"
server_key_version: "encryptionkey@0"
I20260812 07:05:11.105649 18614 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 07:05:11.109354 18635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:11.110852 18614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:05:11.111042 18614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/wal
uuid: "cf71694cf8eb4a8d8a8a7a556d2d5967"
format_stamp: "Formatted at 2026-08-12 07:05:11 on dist-test-slave-69rg"
server_key: "f98144e2681c65fd598d85e33d8065e6"
server_key_iv: "98ecbbd26c23f43e6cb9912fe21ab8a6"
server_key_version: "encryptionkey@0"
I20260812 07:05:11.111266 18614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:05:11.125046 18614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:05:11.129244 18614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:05:11.129568 18614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:05:11.130625 18614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:05:11.132066 18614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:05:11.132143 18614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:11.132261 18614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:05:11.132333 18614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:11.168458 18614 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.1:43263
I20260812 07:05:11.168573 18771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.1:43263 every 8 connection(s)
I20260812 07:05:11.169845 18614 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
I20260812 07:05:11.172938 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 18614
I20260812 07:05:11.173171 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance
I20260812 07:05:11.174144 6388 external_mini_cluster.cc:1428] Setting key d3ab6ec842364fd773a7afc917aa4fcc
Aug 12 07:05:11 dist-test-slave-69rg krb5kdc[18489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518311, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:05:11.194518 18774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:11.172626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:41137 (local address 127.6.61.62:44841)
0812 07:05:11.173207 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:11.173215 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:11.174245 (+ 1030us) server_negotiation.cc:407] Connection header received
0812 07:05:11.176557 (+ 2312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:11.176561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:11.176630 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:11.176723 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:11.178897 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:11.179639 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:11.182173 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:11.182574 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:11.182693 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:11.187408 (+ 4715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:11.187413 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:11.187417 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:05:11.187451 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:11.190969 (+ 3518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:11.191722 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:11.191730 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:11.191736 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:11.191898 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:11.192316 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:11.192323 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:11.192330 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:11.192740 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:05:11.192915 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:11.193702 (+ 787us) server_negotiation.cc:299] Negotiation successful
0812 07:05:11.193998 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":259,"threads_started":1}
I20260812 07:05:11.195192 18775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:11.173138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:44841 (local address 127.6.61.1:41137)
0812 07:05:11.174062 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:11.174114 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:11.176231 (+ 2117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:11.177019 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:11.177031 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:11.177853 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:05:11.178690 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:11.178719 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:11.179817 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:11.179826 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:11.181961 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:11.181979 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:11.182833 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:11.182843 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:11.183175 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:11.183965 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:05:11.184000 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:05:11.187137 (+ 3137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:05:11.191127 (+ 3990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:05:11.191134 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:05:11.191148 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:05:11.191580 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:05:11.192008 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:05:11.192011 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:05:11.192013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:05:11.192192 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:05:11.192924 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:05:11.192931 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:05:11.193271 (+ 340us) client_negotiation.cc:769] Sending connection context
0812 07:05:11.193531 (+ 260us) client_negotiation.cc:240] Negotiation successful
0812 07:05:11.194046 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":316,"threads_started":1}
I20260812 07:05:11.196941 18772 heartbeater.cc:347] Connected to a master server at 127.6.61.62:44841
I20260812 07:05:11.197345 18772 heartbeater.cc:464] Registering TS with master...
I20260812 07:05:11.198321 18772 heartbeater.cc:511] Master 127.6.61.62:44841 requested a full tablet report, sending...
I20260812 07:05:11.200755 18538 ts_manager.cc:194] Registered new tserver with Master: cf71694cf8eb4a8d8a8a7a556d2d5967 (127.6.61.1:43263)
I20260812 07:05:11.202476 18538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.1@KRBTEST.COM'} at 127.6.61.1:41137
WARNING: no policy specified for kudu/127.6.61.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
I20260812 07:05:11.245067 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.2:0
--local_ip_for_outbound_sockets=127.6.61.2
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:44841
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:42463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:05:11.588882 18778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:05:11.589232 18778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:05:11.589406 18778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:05:11.596990 18778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:05:11.597150 18778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:05:11.597344 18778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.2
I20260812 07:05:11.607530 18778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:42463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:44841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.2
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:05:11.608978 18778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:05:11.610435 18778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:05:11.624400 18786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:11.624379 18789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:11.624457 18787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:05:11.625339 18778 server_base.cc:1034] running on GCE node
I20260812 07:05:11.626076 18778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:05:11.627164 18778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:05:11.628434 18778 hybrid_clock.cc:648] HybridClock initialized: now 1786518311628355 us; error 70 us; skew 500 ppm
I20260812 07:05:11.628829 18778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:05:11 dist-test-slave-69rg krb5kdc[18489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518311, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:05:11.632496 18778 init.cc:377] Logged in from keytab as kudu/127.6.61.2@KRBTEST.COM (short username kudu)
I20260812 07:05:11.634572 18778 webserver.cc:460] Webserver started at http://127.6.61.2:35693/ using document root <none> and password file <none>
I20260812 07:05:11.635349 18778 fs_manager.cc:359] Metadata directory not provided
I20260812 07:05:11.635470 18778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:05:11.635758 18778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:05:11.639665 18778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/data/instance:
uuid: "4b3a0376702641c0b3a229a877649eba"
format_stamp: "Formatted at 2026-08-12 07:05:11 on dist-test-slave-69rg"
server_key: "89df6786eeebf62d98aebeaca5f25690"
server_key_iv: "e7a8eff2e9f4645c92551c9e6351789a"
server_key_version: "encryptionkey@0"
I20260812 07:05:11.640508 18778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance:
uuid: "4b3a0376702641c0b3a229a877649eba"
format_stamp: "Formatted at 2026-08-12 07:05:11 on dist-test-slave-69rg"
server_key: "89df6786eeebf62d98aebeaca5f25690"
server_key_iv: "e7a8eff2e9f4645c92551c9e6351789a"
server_key_version: "encryptionkey@0"
I20260812 07:05:11.646217 18778 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 07:05:11.649994 18800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:11.651463 18778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:05:11.651739 18778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/wal
uuid: "4b3a0376702641c0b3a229a877649eba"
format_stamp: "Formatted at 2026-08-12 07:05:11 on dist-test-slave-69rg"
server_key: "89df6786eeebf62d98aebeaca5f25690"
server_key_iv: "e7a8eff2e9f4645c92551c9e6351789a"
server_key_version: "encryptionkey@0"
I20260812 07:05:11.651952 18778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:05:11.668560 18778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:05:11.673237 18778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:05:11.673542 18778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:05:11.674638 18778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:05:11.676041 18778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:05:11.676126 18778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:11.676301 18778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:05:11.676366 18778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:11.724071 18778 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.2:45581
I20260812 07:05:11.724098 18929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.2:45581 every 8 connection(s)
I20260812 07:05:11.725391 18778 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
I20260812 07:05:11.727013 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 18778
I20260812 07:05:11.727245 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance
I20260812 07:05:11.727900 6388 external_mini_cluster.cc:1428] Setting key a3f54dacc4c1dc07b28494868fd87cba
Aug 12 07:05:11 dist-test-slave-69rg krb5kdc[18489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518311, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:05:11.766688 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:11.730499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:51713 (local address 127.6.61.62:44841)
0812 07:05:11.746875 (+ 16376us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:11.746883 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:11.748795 (+ 1912us) server_negotiation.cc:407] Connection header received
0812 07:05:11.751135 (+ 2340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:11.751141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:11.751245 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:11.751381 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:11.753070 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:11.753881 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:11.755620 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:11.756234 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:11.756367 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:11.760252 (+ 3885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:11.760257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:11.760260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:05:11.760296 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:11.763069 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:11.763862 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:11.763870 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:11.763877 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:11.764023 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:11.764480 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:11.764485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:11.764488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:11.764767 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:05:11.764947 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:11.766014 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0812 07:05:11.766245 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16178,"thread_start_us":16015,"threads_started":1}
I20260812 07:05:11.767684 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:11.731140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:44841 (local address 127.6.61.2:51713)
0812 07:05:11.747480 (+ 16340us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:11.747555 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:11.750799 (+ 3244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:11.751675 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:11.751685 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:11.752243 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:05:11.752872 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:11.752890 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:11.754050 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:11.754058 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:11.755473 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:11.755487 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:11.756479 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:11.756487 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:11.756771 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:11.757595 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:05:11.757625 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:05:11.760009 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:05:11.763229 (+ 3220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:05:11.763240 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:05:11.763259 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:05:11.763708 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:05:11.764147 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:05:11.764152 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:05:11.764156 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:05:11.764359 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:05:11.764918 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:05:11.764926 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:05:11.765299 (+ 373us) client_negotiation.cc:769] Sending connection context
0812 07:05:11.765610 (+ 311us) client_negotiation.cc:240] Negotiation successful
0812 07:05:11.766238 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15886,"thread_start_us":15623,"threads_started":1}
I20260812 07:05:11.769263 18930 heartbeater.cc:347] Connected to a master server at 127.6.61.62:44841
I20260812 07:05:11.769629 18930 heartbeater.cc:464] Registering TS with master...
I20260812 07:05:11.770577 18930 heartbeater.cc:511] Master 127.6.61.62:44841 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.61.3@KRBTEST.COM; defaulting to no policy
I20260812 07:05:11.772315 18538 ts_manager.cc:194] Registered new tserver with Master: 4b3a0376702641c0b3a229a877649eba (127.6.61.2:45581)
I20260812 07:05:11.773181 18538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.2@KRBTEST.COM'} at 127.6.61.2:51713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
I20260812 07:05:11.818186 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.3:0
--local_ip_for_outbound_sockets=127.6.61.3
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:44841
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:42463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:05:12.206166 18772 heartbeater.cc:503] Master 127.6.61.62:44841 was elected leader, sending a full tablet report...
W20260812 07:05:12.213323 18938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:05:12.213680 18938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:05:12.213923 18938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:05:12.222393 18938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:05:12.222561 18938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:05:12.222769 18938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.3
I20260812 07:05:12.233300 18938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:42463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:44841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.3
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:05:12.234910 18938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:05:12.236341 18938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:05:12.250331 18944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:12.250463 18946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:12.250487 18951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:05:12.251559 18938 server_base.cc:1034] running on GCE node
I20260812 07:05:12.252243 18938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:05:12.253280 18938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:05:12.254532 18938 hybrid_clock.cc:648] HybridClock initialized: now 1786518312254473 us; error 50 us; skew 500 ppm
I20260812 07:05:12.254907 18938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:05:12 dist-test-slave-69rg krb5kdc[18489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518312, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:05:12.258550 18938 init.cc:377] Logged in from keytab as kudu/127.6.61.3@KRBTEST.COM (short username kudu)
I20260812 07:05:12.260160 18938 webserver.cc:460] Webserver started at http://127.6.61.3:41825/ using document root <none> and password file <none>
I20260812 07:05:12.260876 18938 fs_manager.cc:359] Metadata directory not provided
I20260812 07:05:12.261013 18938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:05:12.261345 18938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:05:12.264588 18938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/data/instance:
uuid: "f5eef038a6cc48659a8e20941edc5201"
format_stamp: "Formatted at 2026-08-12 07:05:12 on dist-test-slave-69rg"
server_key: "a25bfdcb434754dee7bac8c85cbe081a"
server_key_iv: "1948068168f82ad318f37a5e97246198"
server_key_version: "encryptionkey@0"
I20260812 07:05:12.265369 18938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance:
uuid: "f5eef038a6cc48659a8e20941edc5201"
format_stamp: "Formatted at 2026-08-12 07:05:12 on dist-test-slave-69rg"
server_key: "a25bfdcb434754dee7bac8c85cbe081a"
server_key_iv: "1948068168f82ad318f37a5e97246198"
server_key_version: "encryptionkey@0"
I20260812 07:05:12.271665 18938 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260812 07:05:12.275451 18959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:12.277014 18938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:05:12.277248 18938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/wal
uuid: "f5eef038a6cc48659a8e20941edc5201"
format_stamp: "Formatted at 2026-08-12 07:05:12 on dist-test-slave-69rg"
server_key: "a25bfdcb434754dee7bac8c85cbe081a"
server_key_iv: "1948068168f82ad318f37a5e97246198"
server_key_version: "encryptionkey@0"
I20260812 07:05:12.277463 18938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:05:12.305167 18938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:05:12.310264 18938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:05:12.310616 18938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:05:12.311793 18938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:05:12.313333 18938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:05:12.313416 18938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:12.313565 18938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:05:12.313659 18938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:12.352938 18938 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.3:43323
I20260812 07:05:12.352989 19087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.3:43323 every 8 connection(s)
I20260812 07:05:12.354439 18938 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
I20260812 07:05:12.362744 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 18938
I20260812 07:05:12.362978 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance
I20260812 07:05:12.363409 6388 external_mini_cluster.cc:1428] Setting key 8871d7e1696d7ef4cd90e2e276942230
Aug 12 07:05:12 dist-test-slave-69rg krb5kdc[18489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518312, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:05:12.376590 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:12.357381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:36593 (local address 127.6.61.62:44841)
0812 07:05:12.357883 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:12.357888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:12.358961 (+ 1073us) server_negotiation.cc:407] Connection header received
0812 07:05:12.360857 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:12.360862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:12.360956 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:12.361080 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:12.362860 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:12.363791 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:12.365731 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:12.366368 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:12.366509 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:12.370532 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:12.370539 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:12.370544 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:05:12.370594 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:12.373311 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:12.374214 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:12.374217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:12.374220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:12.374338 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:12.374799 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:12.374803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:12.374805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:12.375071 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:05:12.375242 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:12.375950 (+ 708us) server_negotiation.cc:299] Negotiation successful
0812 07:05:12.376193 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":218,"threads_started":1}
I20260812 07:05:12.377630 19091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:12.358034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:44841 (local address 127.6.61.3:36593)
0812 07:05:12.358804 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:12.358842 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:12.360560 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:12.361339 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:12.361348 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:12.362011 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:05:12.362673 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:12.362694 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:12.363961 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:12.363965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:12.365589 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:12.365603 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:12.366605 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:12.366612 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:12.366861 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:12.367601 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:05:12.367626 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:05:12.370272 (+ 2646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:05:12.373466 (+ 3194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:05:12.373477 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:05:12.373496 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:05:12.374031 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:05:12.374455 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:05:12.374459 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:05:12.374463 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:05:12.374668 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:05:12.375216 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:05:12.375224 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:05:12.375552 (+ 328us) client_negotiation.cc:769] Sending connection context
0812 07:05:12.375845 (+ 293us) client_negotiation.cc:240] Negotiation successful
0812 07:05:12.376401 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":303,"threads_started":1}
I20260812 07:05:12.379264 19088 heartbeater.cc:347] Connected to a master server at 127.6.61.62:44841
I20260812 07:05:12.379588 19088 heartbeater.cc:464] Registering TS with master...
I20260812 07:05:12.380336 19088 heartbeater.cc:511] Master 127.6.61.62:44841 requested a full tablet report, sending...
I20260812 07:05:12.381981 18538 ts_manager.cc:194] Registered new tserver with Master: f5eef038a6cc48659a8e20941edc5201 (127.6.61.3:43323)
I20260812 07:05:12.382895 18538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.3@KRBTEST.COM'} at 127.6.61.3:36593
I20260812 07:05:12.392612 6388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 07:05:12.411412 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:12.397887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42870 (local address 127.6.61.62:44841)
0812 07:05:12.398150 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:12.398161 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:12.398695 (+ 534us) server_negotiation.cc:407] Connection header received
0812 07:05:12.398770 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:12.398774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:12.398841 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:12.398927 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:12.400531 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:12.401348 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:12.402796 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:12.403227 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:12.403335 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:12.405738 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:12.405743 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:12.405747 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:05:12.405781 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:12.408538 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:12.409246 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:12.409251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:12.409253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:12.409384 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:12.409809 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:12.409813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:12.409881 (+ 68us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:12.410248 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:05:12.410443 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:12.410807 (+ 364us) server_negotiation.cc:299] Negotiation successful
0812 07:05:12.411011 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 07:05:12.416594 18538 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:05:12.424302 18538 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:05:12.459825 19107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:12.438094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43690 (local address 127.6.61.2:45581)
0812 07:05:12.452018 (+ 13924us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:12.452030 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:12.452065 (+ 35us) server_negotiation.cc:407] Connection header received
0812 07:05:12.452182 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:12.452202 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:12.452440 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:12.452642 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:05:12.453293 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:12.454858 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:12.457112 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:12.458348 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:12.458474 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:12.458671 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:12.459307 (+ 636us) server_negotiation.cc:299] Negotiation successful
0812 07:05:12.459473 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13763,"thread_start_us":13634,"threads_started":1}
I20260812 07:05:12.459904 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:12.437866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:45581 (local address 127.0.0.1:43690)
0812 07:05:12.450186 (+ 12320us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:12.450219 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:12.450355 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:12.452698 (+ 2343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:12.452705 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:12.452737 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:05:12.453084 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:12.453096 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:12.455001 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:12.455007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:12.456943 (+ 1936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:12.456959 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:12.458818 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:12.458824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:12.459042 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:12.459100 (+ 58us) client_negotiation.cc:769] Sending connection context
0812 07:05:12.459256 (+ 156us) client_negotiation.cc:240] Negotiation successful
0812 07:05:12.459458 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12180,"mutex_wait_us":6,"thread_start_us":785,"threads_started":1}
I20260812 07:05:12.460556 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:12.438376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56612 (local address 127.6.61.1:43263)
0812 07:05:12.450047 (+ 11671us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:12.450061 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:12.450102 (+ 41us) server_negotiation.cc:407] Connection header received
0812 07:05:12.450265 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:12.450278 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:12.450639 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:12.451279 (+ 640us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:05:12.452290 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:12.454318 (+ 2028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:12.457214 (+ 2896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:12.458806 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:12.458938 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:12.459119 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:12.460016 (+ 897us) server_negotiation.cc:299] Negotiation successful
0812 07:05:12.460183 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11407,"thread_start_us":12511,"threads_started":1}
I20260812 07:05:12.460973 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:12.437596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:43263 (local address 127.0.0.1:56612)
0812 07:05:12.438772 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:12.438839 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:12.439081 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:12.451337 (+ 12256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:12.451349 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:12.451398 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:05:12.451956 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:12.451978 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:12.454477 (+ 2499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:12.454483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:12.457016 (+ 2533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:12.457034 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:12.459285 (+ 2251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:12.459291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:12.459445 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:12.459478 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 07:05:12.459569 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 07:05:12.459723 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"thread_start_us":14520,"threads_started":1}
I20260812 07:05:12.461519 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:12.441932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:43323 (local address 127.0.0.1:51550)
0812 07:05:12.452427 (+ 10495us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:12.452447 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:12.452589 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:12.453220 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:12.453224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:12.453248 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:05:12.453651 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:12.453661 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:12.455696 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:12.455701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:12.458127 (+ 2426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:12.458153 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:12.460415 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:12.460421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:12.460604 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:12.460639 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 07:05:12.460726 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 07:05:12.460853 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10353,"mutex_wait_us":49,"thread_start_us":7957,"threads_started":1}
I20260812 07:05:12.462070 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:12.442699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51550 (local address 127.6.61.3:43323)
0812 07:05:12.448172 (+ 5473us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:12.448188 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:12.452585 (+ 4397us) server_negotiation.cc:407] Connection header received
0812 07:05:12.452826 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:12.452835 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:12.453021 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:12.453222 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:05:12.453925 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:12.455512 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:12.458329 (+ 2817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:12.459774 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:12.461000 (+ 1226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:12.461332 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:12.461412 (+ 80us) server_negotiation.cc:299] Negotiation successful
0812 07:05:12.461628 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5232,"thread_start_us":5035,"threads_started":1}
I20260812 07:05:12.464570 18695 tablet_service.cc:1467] Processing CreateTablet for tablet b763f7bdd19944ed8b24b2ade3efd92e (DEFAULT_TABLE table=test-table [id=8ee6ced9df2b4460997ea77cb19c0d9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:05:12.464480 18859 tablet_service.cc:1467] Processing CreateTablet for tablet b763f7bdd19944ed8b24b2ade3efd92e (DEFAULT_TABLE table=test-table [id=8ee6ced9df2b4460997ea77cb19c0d9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:05:12.465139 19017 tablet_service.cc:1467] Processing CreateTablet for tablet b763f7bdd19944ed8b24b2ade3efd92e (DEFAULT_TABLE table=test-table [id=8ee6ced9df2b4460997ea77cb19c0d9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:05:12.465993 18695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b763f7bdd19944ed8b24b2ade3efd92e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:05:12.465996 18859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b763f7bdd19944ed8b24b2ade3efd92e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:05:12.466382 19017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b763f7bdd19944ed8b24b2ade3efd92e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:05:12.479130 19113 tablet_bootstrap.cc:492] T b763f7bdd19944ed8b24b2ade3efd92e P f5eef038a6cc48659a8e20941edc5201: Bootstrap starting.
I20260812 07:05:12.481565 19116 tablet_bootstrap.cc:492] T b763f7bdd19944ed8b24b2ade3efd92e P 4b3a0376702641c0b3a229a877649eba: Bootstrap starting.
I20260812 07:05:12.482498 19115 tablet_bootstrap.cc:492] T b763f7bdd19944ed8b24b2ade3efd92e P cf71694cf8eb4a8d8a8a7a556d2d5967: Bootstrap starting.
I20260812 07:05:12.483383 19113 tablet_bootstrap.cc:654] T b763f7bdd19944ed8b24b2ade3efd92e P f5eef038a6cc48659a8e20941edc5201: Neither blocks nor log segments found. Creating new log.
I20260812 07:05:12.484814 19113 log.cc:824] T b763f7bdd19944ed8b24b2ade3efd92e P f5eef038a6cc48659a8e20941edc5201: Log is configured to *not* fsync() on all Append() calls
I20260812 07:05:12.486788 19115 tablet_bootstrap.cc:654] T b763f7bdd19944ed8b24b2ade3efd92e P cf71694cf8eb4a8d8a8a7a556d2d5967: Neither blocks nor log segments found. Creating new log.
I20260812 07:05:12.488106 19116 tablet_bootstrap.cc:654] T b763f7bdd19944ed8b24b2ade3efd92e P 4b3a0376702641c0b3a229a877649eba: Neither blocks nor log segments found. Creating new log.
I20260812 07:05:12.488405 19115 log.cc:824] T b763f7bdd19944ed8b24b2ade3efd92e P cf71694cf8eb4a8d8a8a7a556d2d5967: Log is configured to *not* fsync() on all Append() calls
I20260812 07:05:12.489203 19116 log.cc:824] T b763f7bdd19944ed8b24b2ade3efd92e P 4b3a0376702641c0b3a229a877649eba: Log is configured to *not* fsync() on all Append() calls
I20260812 07:05:12.491923 19116 tablet_bootstrap.cc:492] T b763f7bdd19944ed8b24b2ade3efd92e P 4b3a0376702641c0b3a229a877649eba: No bootstrap required, opened a new log
I20260812 07:05:12.492027 19113 tablet_bootstrap.cc:492] T b763f7bdd19944ed8b24b2ade3efd92e P f5eef038a6cc48659a8e20941edc5201: No bootstrap required, opened a new log
I20260812 07:05:12.492039 19115 tablet_bootstrap.cc:492] T b763f7bdd19944ed8b24b2ade3efd92e P cf71694cf8eb4a8d8a8a7a556d2d5967: No bootstrap required, opened a new log
I20260812 07:05:12.492267 19116 ts_tablet_manager.cc:1397] T b763f7bdd19944ed8b24b2ade3efd92e P 4b3a0376702641c0b3a229a877649eba: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.004s
I20260812 07:05:12.492316 19113 ts_tablet_manager.cc:1397] T b763f7bdd19944ed8b24b2ade3efd92e P f5eef038a6cc48659a8e20941edc5201: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20260812 07:05:12.492316 19115 ts_tablet_manager.cc:1397] T b763f7bdd19944ed8b24b2ade3efd92e P cf71694cf8eb4a8d8a8a7a556d2d5967: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260812 07:05:12.500566 19116 raft_consensus.cc:348] T b763f7bdd19944ed8b24b2ade3efd92e P 4b3a0376702641c0b3a229a877649eba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf71694cf8eb4a8d8a8a7a556d2d5967" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 43263 } } peers { permanent_uuid: "4b3a0376702641c0b3a229a877649eba" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45581 } } peers { permanent_uuid: "f5eef038a6cc48659a8e20941edc5201" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 43323 } }
I20260812 07:05:12.500999 19116 raft_consensus.cc:374] T b763f7bdd19944ed8b24b2ade3efd92e P 4b3a0376702641c0b3a229a877649eba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:05:12.501112 19116 raft_consensus.cc:729] T b763f7bdd19944ed8b24b2ade3efd92e P 4b3a0376702641c0b3a229a877649eba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b3a0376702641c0b3a229a877649eba, State: Initialized, Role: FOLLOWER
I20260812 07:05:12.501940 19116 consensus_queue.cc:260] T b763f7bdd19944ed8b24b2ade3efd92e P 4b3a0376702641c0b3a229a877649eba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf71694cf8eb4a8d8a8a7a556d2d5967" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 43263 } } peers { permanent_uuid: "4b3a0376702641c0b3a229a877649eba" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45581 } } peers { permanent_uuid: "f5eef038a6cc48659a8e20941edc5201" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 43323 } }
I20260812 07:05:12.503130 18930 heartbeater.cc:503] Master 127.6.61.62:44841 was elected leader, sending a full tablet report...
I20260812 07:05:12.503453 19113 raft_consensus.cc:348] T b763f7bdd19944ed8b24b2ade3efd92e P f5eef038a6cc48659a8e20941edc5201 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf71694cf8eb4a8d8a8a7a556d2d5967" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 43263 } } peers { permanent_uuid: "4b3a0376702641c0b3a229a877649eba" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45581 } } peers { permanent_uuid: "f5eef038a6cc48659a8e20941edc5201" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 43323 } }
I20260812 07:05:12.503821 19116 ts_tablet_manager.cc:1428] T b763f7bdd19944ed8b24b2ade3efd92e P 4b3a0376702641c0b3a229a877649eba: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260812 07:05:12.503921 19113 raft_consensus.cc:374] T b763f7bdd19944ed8b24b2ade3efd92e P f5eef038a6cc48659a8e20941edc5201 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:05:12.504025 19113 raft_consensus.cc:729] T b763f7bdd19944ed8b24b2ade3efd92e P f5eef038a6cc48659a8e20941edc5201 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5eef038a6cc48659a8e20941edc5201, State: Initialized, Role: FOLLOWER
I20260812 07:05:12.503772 19115 raft_consensus.cc:348] T b763f7bdd19944ed8b24b2ade3efd92e P cf71694cf8eb4a8d8a8a7a556d2d5967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf71694cf8eb4a8d8a8a7a556d2d5967" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 43263 } } peers { permanent_uuid: "4b3a0376702641c0b3a229a877649eba" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45581 } } peers { permanent_uuid: "f5eef038a6cc48659a8e20941edc5201" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 43323 } }
I20260812 07:05:12.504197 19115 raft_consensus.cc:374] T b763f7bdd19944ed8b24b2ade3efd92e P cf71694cf8eb4a8d8a8a7a556d2d5967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:05:12.504288 19115 raft_consensus.cc:729] T b763f7bdd19944ed8b24b2ade3efd92e P cf71694cf8eb4a8d8a8a7a556d2d5967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf71694cf8eb4a8d8a8a7a556d2d5967, State: Initialized, Role: FOLLOWER
I20260812 07:05:12.504667 19113 consensus_queue.cc:260] T b763f7bdd19944ed8b24b2ade3efd92e P f5eef038a6cc48659a8e20941edc5201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf71694cf8eb4a8d8a8a7a556d2d5967" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 43263 } } peers { permanent_uuid: "4b3a0376702641c0b3a229a877649eba" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45581 } } peers { permanent_uuid: "f5eef038a6cc48659a8e20941edc5201" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 43323 } }
I20260812 07:05:12.504844 19115 consensus_queue.cc:260] T b763f7bdd19944ed8b24b2ade3efd92e P cf71694cf8eb4a8d8a8a7a556d2d5967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf71694cf8eb4a8d8a8a7a556d2d5967" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 43263 } } peers { permanent_uuid: "4b3a0376702641c0b3a229a877649eba" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45581 } } peers { permanent_uuid: "f5eef038a6cc48659a8e20941edc5201" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 43323 } }
I20260812 07:05:12.505872 19088 heartbeater.cc:503] Master 127.6.61.62:44841 was elected leader, sending a full tablet report...
I20260812 07:05:12.506790 19113 ts_tablet_manager.cc:1428] T b763f7bdd19944ed8b24b2ade3efd92e P f5eef038a6cc48659a8e20941edc5201: Time spent starting tablet: real 0.014s user 0.011s sys 0.005s
I20260812 07:05:12.509075 19115 ts_tablet_manager.cc:1428] T b763f7bdd19944ed8b24b2ade3efd92e P cf71694cf8eb4a8d8a8a7a556d2d5967: Time spent starting tablet: real 0.017s user 0.008s sys 0.010s
I20260812 07:05:12.596560 19122 raft_consensus.cc:482] T b763f7bdd19944ed8b24b2ade3efd92e P 4b3a0376702641c0b3a229a877649eba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:05:12.596930 19122 raft_consensus.cc:504] T b763f7bdd19944ed8b24b2ade3efd92e P 4b3a0376702641c0b3a229a877649eba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf71694cf8eb4a8d8a8a7a556d2d5967" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 43263 } } peers { permanent_uuid: "4b3a0376702641c0b3a229a877649eba" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45581 } } peers { permanent_uuid: "f5eef038a6cc48659a8e20941edc5201" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 43323 } }
I20260812 07:05:12.598937 19122 leader_election.cc:288] T b763f7bdd19944ed8b24b2ade3efd92e P 4b3a0376702641c0b3a229a877649eba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf71694cf8eb4a8d8a8a7a556d2d5967 (127.6.61.1:43263), f5eef038a6cc48659a8e20941edc5201 (127.6.61.3:43323)
W20260812 07:05:12.606238 19089 tablet.cc:2369] T b763f7bdd19944ed8b24b2ade3efd92e P f5eef038a6cc48659a8e20941edc5201: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:05:12.606840 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:12.599476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:35957 (local address 127.6.61.1:43263)
0812 07:05:12.599743 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:12.599753 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:12.600597 (+ 844us) server_negotiation.cc:407] Connection header received
0812 07:05:12.600901 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:12.600912 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:12.601010 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:12.601107 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:05:12.601956 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:12.602744 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:12.604957 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:12.605769 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:12.605974 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:12.606090 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:12.606403 (+ 313us) server_negotiation.cc:299] Negotiation successful
0812 07:05:12.606553 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 07:05:12.606981 19125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:12.599343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:43263 (local address 127.6.61.2:35957)
0812 07:05:12.600461 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:12.600477 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:12.600680 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:12.601158 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:12.601163 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:12.601190 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:05:12.601767 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:12.601784 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:12.602887 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:12.602893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:12.604768 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:12.604780 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:12.606057 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:12.606063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:12.606232 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:12.606256 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 07:05:12.606339 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 07:05:12.606490 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":941,"mutex_wait_us":60,"spinlock_wait_cycles":25984,"thread_start_us":591,"threads_started":1}
I20260812 07:05:12.607760 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:12.599394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:39455 (local address 127.6.61.3:43323)
0812 07:05:12.599633 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:12.599641 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:12.600757 (+ 1116us) server_negotiation.cc:407] Connection header received
0812 07:05:12.601015 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:12.601020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:12.601105 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:12.601190 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:05:12.602110 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:12.602887 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:12.604978 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:12.605770 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:12.605976 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:12.606098 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:12.607395 (+ 1297us) server_negotiation.cc:299] Negotiation successful
0812 07:05:12.607542 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 07:05:12.607779 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:12.599455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:43323 (local address 127.6.61.2:39455)
0812 07:05:12.600619 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:12.600646 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:12.600799 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:12.601327 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:12.601335 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:12.601373 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:05:12.601933 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:12.601948 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:12.603065 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:12.603069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:12.604780 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:12.604794 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:12.606692 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:12.606699 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:12.607191 (+ 492us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:12.607249 (+ 58us) client_negotiation.cc:769] Sending connection context
0812 07:05:12.607350 (+ 101us) client_negotiation.cc:240] Negotiation successful
0812 07:05:12.607441 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1007,"mutex_wait_us":23,"thread_start_us":646,"threads_started":1}
I20260812 07:05:12.608239 18719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b763f7bdd19944ed8b24b2ade3efd92e" candidate_uuid: "4b3a0376702641c0b3a229a877649eba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf71694cf8eb4a8d8a8a7a556d2d5967" is_pre_election: true
I20260812 07:05:12.608765 19038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b763f7bdd19944ed8b24b2ade3efd92e" candidate_uuid: "4b3a0376702641c0b3a229a877649eba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5eef038a6cc48659a8e20941edc5201" is_pre_election: true
I20260812 07:05:12.608872 18719 raft_consensus.cc:2454] T b763f7bdd19944ed8b24b2ade3efd92e P cf71694cf8eb4a8d8a8a7a556d2d5967 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b3a0376702641c0b3a229a877649eba in term 0.
I20260812 07:05:12.609277 19038 raft_consensus.cc:2454] T b763f7bdd19944ed8b24b2ade3efd92e P f5eef038a6cc48659a8e20941edc5201 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b3a0376702641c0b3a229a877649eba in term 0.
I20260812 07:05:12.609738 18802 leader_election.cc:302] T b763f7bdd19944ed8b24b2ade3efd92e P 4b3a0376702641c0b3a229a877649eba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b3a0376702641c0b3a229a877649eba, cf71694cf8eb4a8d8a8a7a556d2d5967; no voters:
I20260812 07:05:12.610220 19122 raft_consensus.cc:2781] T b763f7bdd19944ed8b24b2ade3efd92e P 4b3a0376702641c0b3a229a877649eba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:05:12.610396 19122 raft_consensus.cc:482] T b763f7bdd19944ed8b24b2ade3efd92e P 4b3a0376702641c0b3a229a877649eba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:05:12.610509 19122 raft_consensus.cc:3037] T b763f7bdd19944ed8b24b2ade3efd92e P 4b3a0376702641c0b3a229a877649eba [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:05:12.612811 19122 raft_consensus.cc:504] T b763f7bdd19944ed8b24b2ade3efd92e P 4b3a0376702641c0b3a229a877649eba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf71694cf8eb4a8d8a8a7a556d2d5967" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 43263 } } peers { permanent_uuid: "4b3a0376702641c0b3a229a877649eba" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45581 } } peers { permanent_uuid: "f5eef038a6cc48659a8e20941edc5201" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 43323 } }
I20260812 07:05:12.613445 19122 leader_election.cc:288] T b763f7bdd19944ed8b24b2ade3efd92e P 4b3a0376702641c0b3a229a877649eba [CANDIDATE]: Term 1 election: Requested vote from peers cf71694cf8eb4a8d8a8a7a556d2d5967 (127.6.61.1:43263), f5eef038a6cc48659a8e20941edc5201 (127.6.61.3:43323)
I20260812 07:05:12.613966 19038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b763f7bdd19944ed8b24b2ade3efd92e" candidate_uuid: "4b3a0376702641c0b3a229a877649eba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5eef038a6cc48659a8e20941edc5201"
I20260812 07:05:12.613979 18719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b763f7bdd19944ed8b24b2ade3efd92e" candidate_uuid: "4b3a0376702641c0b3a229a877649eba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf71694cf8eb4a8d8a8a7a556d2d5967"
I20260812 07:05:12.614164 19038 raft_consensus.cc:3037] T b763f7bdd19944ed8b24b2ade3efd92e P f5eef038a6cc48659a8e20941edc5201 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:05:12.614212 18719 raft_consensus.cc:3037] T b763f7bdd19944ed8b24b2ade3efd92e P cf71694cf8eb4a8d8a8a7a556d2d5967 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:05:12.616503 19038 raft_consensus.cc:2454] T b763f7bdd19944ed8b24b2ade3efd92e P f5eef038a6cc48659a8e20941edc5201 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b3a0376702641c0b3a229a877649eba in term 1.
I20260812 07:05:12.616962 18801 leader_election.cc:302] T b763f7bdd19944ed8b24b2ade3efd92e P 4b3a0376702641c0b3a229a877649eba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b3a0376702641c0b3a229a877649eba, f5eef038a6cc48659a8e20941edc5201; no voters:
I20260812 07:05:12.616953 18719 raft_consensus.cc:2454] T b763f7bdd19944ed8b24b2ade3efd92e P cf71694cf8eb4a8d8a8a7a556d2d5967 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b3a0376702641c0b3a229a877649eba in term 1.
I20260812 07:05:12.617301 19122 raft_consensus.cc:2781] T b763f7bdd19944ed8b24b2ade3efd92e P 4b3a0376702641c0b3a229a877649eba [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:05:12.617870 19122 raft_consensus.cc:686] T b763f7bdd19944ed8b24b2ade3efd92e P 4b3a0376702641c0b3a229a877649eba [term 1 LEADER]: Becoming Leader. State: Replica: 4b3a0376702641c0b3a229a877649eba, State: Running, Role: LEADER
I20260812 07:05:12.618484 19122 consensus_queue.cc:237] T b763f7bdd19944ed8b24b2ade3efd92e P 4b3a0376702641c0b3a229a877649eba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf71694cf8eb4a8d8a8a7a556d2d5967" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 43263 } } peers { permanent_uuid: "4b3a0376702641c0b3a229a877649eba" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45581 } } peers { permanent_uuid: "f5eef038a6cc48659a8e20941edc5201" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 43323 } }
I20260812 07:05:12.623195 18538 catalog_manager.cc:5495] T b763f7bdd19944ed8b24b2ade3efd92e P 4b3a0376702641c0b3a229a877649eba reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b3a0376702641c0b3a229a877649eba (127.6.61.2). New cstate: current_term: 1 leader_uuid: "4b3a0376702641c0b3a229a877649eba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf71694cf8eb4a8d8a8a7a556d2d5967" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 43263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b3a0376702641c0b3a229a877649eba" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5eef038a6cc48659a8e20941edc5201" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 43323 } health_report { overall_health: UNKNOWN } } }
I20260812 07:05:12.651371 19107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:12.646046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43704 (local address 127.6.61.2:45581)
0812 07:05:12.646307 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:12.646313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:12.646335 (+ 22us) server_negotiation.cc:407] Connection header received
0812 07:05:12.646438 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:12.646443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:12.646543 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:12.646699 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:05:12.647088 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:12.647821 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:12.649170 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:12.649579 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:12.649713 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:12.650286 (+ 573us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:05:12.650784 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:05:12.650945 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:12.651015 (+ 70us) server_negotiation.cc:299] Negotiation successful
0812 07:05:12.651098 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 07:05:12.659785 19038 raft_consensus.cc:1261] T b763f7bdd19944ed8b24b2ade3efd92e P f5eef038a6cc48659a8e20941edc5201 [term 1 FOLLOWER]: Refusing update from remote peer 4b3a0376702641c0b3a229a877649eba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:05:12.660044 18719 raft_consensus.cc:1261] T b763f7bdd19944ed8b24b2ade3efd92e P cf71694cf8eb4a8d8a8a7a556d2d5967 [term 1 FOLLOWER]: Refusing update from remote peer 4b3a0376702641c0b3a229a877649eba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:05:12.660851 19127 consensus_queue.cc:1035] T b763f7bdd19944ed8b24b2ade3efd92e P 4b3a0376702641c0b3a229a877649eba [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5eef038a6cc48659a8e20941edc5201" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 43323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:05:12.661250 19122 consensus_queue.cc:1035] T b763f7bdd19944ed8b24b2ade3efd92e P 4b3a0376702641c0b3a229a877649eba [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf71694cf8eb4a8d8a8a7a556d2d5967" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 43263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 07:05:12.672434 18773 tablet.cc:2369] T b763f7bdd19944ed8b24b2ade3efd92e P cf71694cf8eb4a8d8a8a7a556d2d5967: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:05:12.673930 19130 mvcc.cc:205] Tried to move back new op lower bound from 7317579008644575232 to 7317579008489517056. Current Snapshot: MvccSnapshot[applied={T|T < 7317579008644575232}]
I20260812 07:05:12.688326 18536 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:05:12.688678 18536 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:05:12.692682 18536 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d17d48231e9443c89144db3c3a73b40b: Sending DeleteTablet for 3 replicas of tablet b763f7bdd19944ed8b24b2ade3efd92e
I20260812 07:05:12.694209 18695 tablet_service.cc:1514] Processing DeleteTablet for tablet b763f7bdd19944ed8b24b2ade3efd92e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:05:12 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:56612
I20260812 07:05:12.694429 18859 tablet_service.cc:1514] Processing DeleteTablet for tablet b763f7bdd19944ed8b24b2ade3efd92e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:05:12 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:43690
I20260812 07:05:12.694483 19017 tablet_service.cc:1514] Processing DeleteTablet for tablet b763f7bdd19944ed8b24b2ade3efd92e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:05:12 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:51550
I20260812 07:05:12.695859 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 18614
I20260812 07:05:12.705220 19145 tablet_replica.cc:330] T b763f7bdd19944ed8b24b2ade3efd92e P 4b3a0376702641c0b3a229a877649eba: stopping tablet replica
I20260812 07:05:12.705886 19145 raft_consensus.cc:2229] T b763f7bdd19944ed8b24b2ade3efd92e P 4b3a0376702641c0b3a229a877649eba [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:05:12.706919 19145 raft_consensus.cc:2258] T b763f7bdd19944ed8b24b2ade3efd92e P 4b3a0376702641c0b3a229a877649eba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:05:12.709349 19144 tablet_replica.cc:330] T b763f7bdd19944ed8b24b2ade3efd92e P cf71694cf8eb4a8d8a8a7a556d2d5967: stopping tablet replica
I20260812 07:05:12.709890 19144 raft_consensus.cc:2229] T b763f7bdd19944ed8b24b2ade3efd92e P cf71694cf8eb4a8d8a8a7a556d2d5967 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:05:12.710454 19144 raft_consensus.cc:2258] T b763f7bdd19944ed8b24b2ade3efd92e P cf71694cf8eb4a8d8a8a7a556d2d5967 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:05:12.711316 19146 tablet_replica.cc:330] T b763f7bdd19944ed8b24b2ade3efd92e P f5eef038a6cc48659a8e20941edc5201: stopping tablet replica
I20260812 07:05:12.711755 19146 raft_consensus.cc:2229] T b763f7bdd19944ed8b24b2ade3efd92e P f5eef038a6cc48659a8e20941edc5201 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:05:12.712289 19146 raft_consensus.cc:2258] T b763f7bdd19944ed8b24b2ade3efd92e P f5eef038a6cc48659a8e20941edc5201 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:05:12.712842 19145 ts_tablet_manager.cc:1905] T b763f7bdd19944ed8b24b2ade3efd92e P 4b3a0376702641c0b3a229a877649eba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:05:12.714471 19146 ts_tablet_manager.cc:1905] T b763f7bdd19944ed8b24b2ade3efd92e P f5eef038a6cc48659a8e20941edc5201: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:05:12.716780 19144 ts_tablet_manager.cc:1905] T b763f7bdd19944ed8b24b2ade3efd92e P cf71694cf8eb4a8d8a8a7a556d2d5967: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:05:12.720727 19145 ts_tablet_manager.cc:1918] T b763f7bdd19944ed8b24b2ade3efd92e P 4b3a0376702641c0b3a229a877649eba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:05:12.720918 19145 log.cc:1196] T b763f7bdd19944ed8b24b2ade3efd92e P 4b3a0376702641c0b3a229a877649eba: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-1/wal/wals/b763f7bdd19944ed8b24b2ade3efd92e
I20260812 07:05:12.721557 19145 ts_tablet_manager.cc:1939] T b763f7bdd19944ed8b24b2ade3efd92e P 4b3a0376702641c0b3a229a877649eba: Deleting consensus metadata
I20260812 07:05:12.723779 18520 catalog_manager.cc:4841] TS 4b3a0376702641c0b3a229a877649eba (127.6.61.2:45581): tablet b763f7bdd19944ed8b24b2ade3efd92e (table test-table [id=8ee6ced9df2b4460997ea77cb19c0d9f]) successfully deleted
I20260812 07:05:12.724403 19144 ts_tablet_manager.cc:1918] T b763f7bdd19944ed8b24b2ade3efd92e P cf71694cf8eb4a8d8a8a7a556d2d5967: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:05:12.724545 19144 log.cc:1196] T b763f7bdd19944ed8b24b2ade3efd92e P cf71694cf8eb4a8d8a8a7a556d2d5967: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-0/wal/wals/b763f7bdd19944ed8b24b2ade3efd92e
I20260812 07:05:12.725068 19144 ts_tablet_manager.cc:1939] T b763f7bdd19944ed8b24b2ade3efd92e P cf71694cf8eb4a8d8a8a7a556d2d5967: Deleting consensus metadata
I20260812 07:05:12.726910 18521 catalog_manager.cc:4841] TS cf71694cf8eb4a8d8a8a7a556d2d5967 (127.6.61.1:43263): tablet b763f7bdd19944ed8b24b2ade3efd92e (table test-table [id=8ee6ced9df2b4460997ea77cb19c0d9f]) successfully deleted
Aug 12 07:05:12 dist-test-slave-69rg krb5kdc[18489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.1@KRBTEST.COM
I20260812 07:05:12.728277 19146 ts_tablet_manager.cc:1918] T b763f7bdd19944ed8b24b2ade3efd92e P f5eef038a6cc48659a8e20941edc5201: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:05:12.728459 19146 log.cc:1196] T b763f7bdd19944ed8b24b2ade3efd92e P f5eef038a6cc48659a8e20941edc5201: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518208686283-6388-0/minicluster-data/ts-2/wal/wals/b763f7bdd19944ed8b24b2ade3efd92e
I20260812 07:05:12.729049 19146 ts_tablet_manager.cc:1939] T b763f7bdd19944ed8b24b2ade3efd92e P f5eef038a6cc48659a8e20941edc5201: Deleting consensus metadata
I20260812 07:05:12.731101 18520 catalog_manager.cc:4841] TS f5eef038a6cc48659a8e20941edc5201 (127.6.61.3:43323): tablet b763f7bdd19944ed8b24b2ade3efd92e (table test-table [id=8ee6ced9df2b4460997ea77cb19c0d9f]) successfully deleted
I20260812 07:05:12.736048 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:12.697721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56624 (local address 127.6.61.1:43263)
0812 07:05:12.697995 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:12.698004 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:12.709095 (+ 11091us) server_negotiation.cc:407] Connection header received
0812 07:05:12.709734 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:12.709742 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:12.709892 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:12.709985 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:12.712225 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:12.713384 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:12.715475 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:12.716172 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:12.716287 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:12.729052 (+ 12765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:12.729066 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:12.729083 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:05:12.729140 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:12.732692 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:12.733601 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:12.733612 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:12.733616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:12.733786 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:12.734235 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:12.734241 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:12.734244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:12.734715 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:05:12.735030 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:12.735396 (+ 366us) server_negotiation.cc:299] Negotiation successful
0812 07:05:12.735656 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 07:05:12.736747 18766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:05:12.834666 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 18614
I20260812 07:05:12.858702 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 18778
Aug 12 07:05:12 dist-test-slave-69rg krb5kdc[18489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.2@KRBTEST.COM
I20260812 07:05:12.877508 19107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:12.859547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43720 (local address 127.6.61.2:45581)
0812 07:05:12.859733 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:12.859738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:12.859923 (+ 185us) server_negotiation.cc:407] Connection header received
0812 07:05:12.860106 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:12.860114 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:12.860221 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:12.860335 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:12.861669 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:12.862984 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:12.864305 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:12.864985 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:12.865124 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:12.869896 (+ 4772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:12.869915 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:12.869938 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:05:12.870000 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:12.874233 (+ 4233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:12.875060 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:12.875071 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:12.875075 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:12.875256 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:12.875681 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:12.875686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:12.875691 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:12.876197 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:05:12.876597 (+ 400us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:12.876847 (+ 250us) server_negotiation.cc:299] Negotiation successful
0812 07:05:12.877096 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 07:05:12.878449 18923 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:05:12.996961 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 18778
I20260812 07:05:13.022941 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 18938
Aug 12 07:05:13 dist-test-slave-69rg krb5kdc[18489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.3@KRBTEST.COM
I20260812 07:05:13.038882 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:13.023751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51558 (local address 127.6.61.3:43323)
0812 07:05:13.023927 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:13.023932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:13.024042 (+ 110us) server_negotiation.cc:407] Connection header received
0812 07:05:13.024157 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:13.024162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:13.024241 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:13.024358 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:13.025677 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:13.026509 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:13.027860 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:13.028253 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:13.028351 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:13.032820 (+ 4469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:13.032837 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:13.032855 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:05:13.032899 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:13.035722 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:13.036479 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:13.036487 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:13.036490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:13.036653 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:13.037169 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:13.037175 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:13.037179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:13.037545 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:05:13.037892 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:13.038276 (+ 384us) server_negotiation.cc:299] Negotiation successful
0812 07:05:13.038467 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 07:05:13.039546 19082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:05:13.131685 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 18938
I20260812 07:05:13.154543 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 18503
I20260812 07:05:13.155992 18576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:05:13.268936 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 18503
2026-08-12T07:05:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4797 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:05:13 dist-test-slave-69rg krb5kdc[19154](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:05:13 dist-test-slave-69rg krb5kdc[19154](info): set up 2 sockets
Aug 12 07:05:13 dist-test-slave-69rg krb5kdc[19154](info): commencing operation
krb5kdc: starting...
W20260812 07:05:14.479151 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.004s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:05:14 dist-test-slave-69rg krb5kdc[19154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:05:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:05:14Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.61.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.61.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.61.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/rangeradmin_127.6.61.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.61.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/rangeradmin_127.6.61.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.61.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.61.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.61.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/rangerlookup_127.6.61.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.61.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/rangerlookup_127.6.61.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.61.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.61.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.61.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/HTTP_127.6.61.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.61.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/HTTP_127.6.61.20@KRBTEST.COM.keytab.
I20260812 07:05:14.908979 6388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task6k_OM2/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/postgres -l logfile start
2026-08-12 07:05:17.677 UTC [19188] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 07:05:17.677 UTC [19188] LOG: listening on IPv4 address "127.6.61.20", port 52539
2026-08-12 07:05:17.681 UTC [19188] LOG: listening on Unix socket "/tmp/.s.PGSQL.52539"
2026-08-12 07:05:17.703 UTC [19191] LOG: database system was shut down at 2026-08-12 07:05:16 UTC
2026-08-12 07:05:17.707 UTC [19188] LOG: database system is ready to accept connections
I20260812 07:05:18.792618 6388 mini_postgres.cc:96] Postgres bound to 52539
2026-08-12 07:05:18.817 UTC [19201] FATAL: database "slave" does not exist
127.6.61.20:52539 - accepting connections
I20260812 07:05:18.818377 6388 mini_ranger.cc:161] Starting Ranger...
I20260812 07:05:18.851069 6388 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 07:05:19.440670 6388 mini_ranger.cc:87] Created ranger Postgres database
I20260812 07:05:19.440838 6388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ranger-admin
2026-08-12 07:05:19,750 [I] DB FLAVOR :POSTGRES
2026-08-12 07:05:19,751 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 07:05:19,751 [I] Checking connection..
2026-08-12 07:05:19,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:52539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 07:05:20,020 [I] Checking connection passed.
2026-08-12 07:05:20,021 [I] --------- Verifying version history table ---------
2026-08-12 07:05:20,021 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:52539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 07:05:20,304 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 07:05:20,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:52539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 07:05:20,565 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 07:05:20,565 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 07:05:20,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:52539/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 07:05:20.831 UTC [19280] WARNING: there is no transaction in progress
2026-08-12 07:05:20,839 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 07:05:20,839 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:52539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 07:05:21,109 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 07:05:21,109 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 07:05:21,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:52539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 07:05:21,372 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:52539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 07:05:21,637 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:52539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-69rg.c.gcp-upstream.internal','N') ;"
2026-08-12 07:05:21,902 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:52539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 07:05:22,165 [I] Table x_portal_user does not exist in database ranger
2026-08-12 07:05:22,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:52539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 07:05:22,435 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 07:05:22,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:52539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 07:05:22,691 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:52539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 07:05:22,947 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 07:05:22,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:52539/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 07:05:23.964 UTC [19416] WARNING: there is no transaction in progress
2026-08-12 07:05:23.983 UTC [19416] WARNING: there is no transaction in progress
2026-08-12 07:05:24.002 UTC [19416] WARNING: there is no transaction in progress
2026-08-12 07:05:24.020 UTC [19416] WARNING: there is no transaction in progress
2026-08-12 07:05:24.037 UTC [19416] WARNING: there is no transaction in progress
2026-08-12 07:05:24.057 UTC [19416] WARNING: there is no transaction in progress
2026-08-12 07:05:24.125 UTC [19416] WARNING: there is no transaction in progress
2026-08-12 07:05:24.132 UTC [19416] WARNING: there is no transaction in progress
2026-08-12 07:05:24.146 UTC [19416] WARNING: there is no transaction in progress
2026-08-12 07:05:24.158 UTC [19416] WARNING: there is no transaction in progress
2026-08-12 07:05:24.170 UTC [19416] WARNING: there is no transaction in progress
2026-08-12 07:05:24.185 UTC [19416] WARNING: there is no transaction in progress
2026-08-12 07:05:24.195 UTC [19416] WARNING: there is no transaction in progress
2026-08-12 07:05:24.203 UTC [19416] WARNING: there is no transaction in progress
2026-08-12 07:05:24.211 UTC [19416] WARNING: there is no transaction in progress
2026-08-12 07:05:24.218 UTC [19416] WARNING: there is no transaction in progress
2026-08-12 07:05:24,943 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 07:05:24,943 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:52539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 07:05:25,198 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 07:05:25,199 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:52539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 07:05:25,454 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 07:05:25,454 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:52539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 07:05:25,717 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 07:05:25,718 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:52539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 07:05:25,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:52539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 07:05:26,237 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:52539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-69rg.c.gcp-upstream.internal';"
2026-08-12 07:05:26,498 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 07:05:26,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:52539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 07:05:26,745 [I] DB_PATCHES have already been applied
I20260812 07:05:26.751930 6388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ranger-admin:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task6k_OM2/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 07:05:26.752072 6388 mini_ranger.cc:193] Using host: 127.6.61.20
I20260812 07:05:26.775135 6388 mini_ranger.cc:227] Ranger admin URL: http://127.6.61.20:37949
Aug 12, 2026 7:05:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 7:05:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 7:05:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 7:05:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 7:05:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 7:05:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 7:05:27 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 07:05:28.017098 6388 mini_ranger.cc:160] Time spent starting Ranger: real 9.199s user 0.001s sys 0.065s
Aug 12, 2026 7:05:48 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 7:05:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 12, 2026 7:05:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 7:05:48 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 7:05:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 7:05:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 7:05:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 7:05:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 7:05:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 7:05:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 7:05:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 7:05:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 7:05:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 7:05:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 7:05:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 7:05:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 7:05:48 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 7:05:49 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 07:05:50.008478 6388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.61.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.61.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.61.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/rangerkms_127.6.61.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.61.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/rangerkms_127.6.61.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.61.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.61.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.61.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/HTTP_127.6.61.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.61.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/HTTP_127.6.61.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 07:05:50 dist-test-slave-69rg krb5kdc[19154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518350, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 07:05:50.258450 6388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 07:05:50.294087 6388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 07:05:51.347841 6388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 07:05:51.348049 6388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ranger-kms
I20260812 07:05:51.348251 6388 mini_ranger_kms.cc:227] Using postgres at 127.6.61.20:52539
2026-08-12 07:05:52,014 [I] DB FLAVOR :POSTGRES
2026-08-12 07:05:52,014 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 07:05:52,014 [I] Checking connection
2026-08-12 07:05:52,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:52539/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 07:05:52,293 [I] connection success
2026-08-12 07:05:52,293 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 07:05:52,293 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 07:05:52,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:52539/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 07:05:52,556 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 07:05:52,557 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 07:05:52,557 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 07:05:52,557 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:52539/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 07:05:52,824 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 07:05:53.317587 6388 mini_ranger_kms.cc:310] Created kms service
I20260812 07:05:53.544646 6388 mini_ranger_kms.cc:326] Created kudu user
I20260812 07:05:53.592631 6388 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 07:05:53 dist-test-slave-69rg krb5kdc[19154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518350, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.61.20@KRBTEST.COM
I20260812 07:05:54.026418 6388 mini_ranger_kms.cc:384] Added ranger policy
I20260812 07:05:54.028064 6388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ranger-kms:/tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 07:05:54.028152 6388 mini_ranger_kms.cc:240] Using host: 127.6.61.20
I20260812 07:05:54.063480 6388 mini_ranger_kms.cc:276] Ranger KMS PID: 19637
I20260812 07:05:54.063889 6388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.61.20:51595
Aug 12, 2026 7:05:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 7:05:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 7:05:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 7:05:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 7:05:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 7:05:55 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 12 07:05:56 dist-test-slave-69rg krb5kdc[19154](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786518356, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.61.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:05:56.638612 6388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.380s user 0.142s sys 0.449s
I20260812 07:05:56.638988 6388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 07:05:56.639205 6388 mini_ranger_kms.cc:401] 127.6.61.20:51595/kms/v1/keys
Aug 12 07:05:56 dist-test-slave-69rg krb5kdc[19154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518356, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.61.20@KRBTEST.COM for HTTP/127.6.61.20@KRBTEST.COM
Aug 12 07:05:57 dist-test-slave-69rg krb5kdc[19154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518356, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.61.20@KRBTEST.COM for HTTP/127.6.61.20@KRBTEST.COM
Aug 12 07:05:59 dist-test-slave-69rg krb5kdc[19154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
I20260812 07:05:59.774206 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:36111
--webserver_interface=127.6.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:43813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:36111
--ranger_config_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.61.20:51595/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:06:00.306898 19694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:00.307184 19694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:00.307276 19694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:00.314724 19694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:06:00.314945 19694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:00.315043 19694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:00.315124 19694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:06:00.315176 19694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:06:00.326699 19694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:43813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.61.20:51595/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:36111
--ranger_config_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:36111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:06:00.328233 19694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:00.329883 19694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:00.344579 19699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:00.344859 19701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:00.345765 19703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:00.346041 19694 server_base.cc:1034] running on GCE node
I20260812 07:06:00.346968 19694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:00.348618 19694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:00.349915 19694 hybrid_clock.cc:648] HybridClock initialized: now 1786518360349790 us; error 35 us; skew 500 ppm
I20260812 07:06:00.350349 19694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:06:00 dist-test-slave-69rg krb5kdc[19154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518360, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:00.354542 19694 init.cc:377] Logged in from keytab as kudu/127.6.61.62@KRBTEST.COM (short username kudu)
I20260812 07:06:00.356500 19694 webserver.cc:460] Webserver started at http://127.6.61.62:40993/ using document root <none> and password file <none>
I20260812 07:06:00.357348 19694 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:00.357537 19694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:00.357967 19694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 07:06:00 dist-test-slave-69rg krb5kdc[19154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518360, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for HTTP/127.6.61.20@KRBTEST.COM
I20260812 07:06:00.416837 19694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "eb4472c41fcc4e2ca6f59256afb88295"
format_stamp: "Formatted at 2026-08-12 07:06:00 on dist-test-slave-69rg"
server_key: "88e2d5075f60697af9bf3a9caa3d7827"
server_key_iv: "9bdec940bd9ebb9cc646fcf3f3e03122"
server_key_version: "kuduclusterkey@0"
I20260812 07:06:00.418139 19694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "eb4472c41fcc4e2ca6f59256afb88295"
format_stamp: "Formatted at 2026-08-12 07:06:00 on dist-test-slave-69rg"
server_key: "88e2d5075f60697af9bf3a9caa3d7827"
server_key_iv: "9bdec940bd9ebb9cc646fcf3f3e03122"
server_key_version: "kuduclusterkey@0"
I20260812 07:06:00.425580 19694 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260812 07:06:00.446241 19712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:00.448040 19694 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260812 07:06:00.448319 19694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "eb4472c41fcc4e2ca6f59256afb88295"
format_stamp: "Formatted at 2026-08-12 07:06:00 on dist-test-slave-69rg"
server_key: "88e2d5075f60697af9bf3a9caa3d7827"
server_key_iv: "9bdec940bd9ebb9cc646fcf3f3e03122"
server_key_version: "kuduclusterkey@0"
I20260812 07:06:00.448652 19694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:00.466131 19694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:00.471714 19694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:00.472229 19694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:00.516074 19694 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:36111
I20260812 07:06:00.516229 19772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:36111 every 8 connection(s)
I20260812 07:06:00.517990 19694 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:06:00.523612 19775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:00.527196 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 19694
I20260812 07:06:00.527448 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:06:00.528345 6388 external_mini_cluster.cc:1428] Setting key a2c8ff2d754a4350d39510b68017520d
I20260812 07:06:00.540941 19775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb4472c41fcc4e2ca6f59256afb88295: Bootstrap starting.
Aug 12 07:06:00 dist-test-slave-69rg krb5kdc[19154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:06:00.546347 19775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb4472c41fcc4e2ca6f59256afb88295: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:00.548259 19775 log.cc:824] T 00000000000000000000000000000000 P eb4472c41fcc4e2ca6f59256afb88295: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:00.552500 19775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb4472c41fcc4e2ca6f59256afb88295: No bootstrap required, opened a new log
I20260812 07:06:00.556368 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:00.530991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39338 (local address 127.6.61.62:36111)
0812 07:06:00.532472 (+ 1481us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:00.532485 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:00.532550 (+ 65us) server_negotiation.cc:407] Connection header received
0812 07:06:00.535200 (+ 2650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:00.535232 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:00.535628 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:00.536029 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:00.537586 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:00.539534 (+ 1948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:00.540839 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:00.541741 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:00.541986 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:00.546855 (+ 4869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:00.546866 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:00.546884 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:00.546927 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:00.551443 (+ 4516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:00.552234 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:00.552241 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:00.552248 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:00.552439 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:00.552961 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:00.552966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:00.552971 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:00.553731 (+ 760us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:00.554204 (+ 473us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:00.554684 (+ 480us) server_negotiation.cc:299] Negotiation successful
0812 07:06:00.555106 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1192,"thread_start_us":834,"threads_started":1}
I20260812 07:06:00.565148 19775 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb4472c41fcc4e2ca6f59256afb88295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb4472c41fcc4e2ca6f59256afb88295" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 36111 } }
I20260812 07:06:00.565740 19775 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb4472c41fcc4e2ca6f59256afb88295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:00.565980 19775 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb4472c41fcc4e2ca6f59256afb88295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb4472c41fcc4e2ca6f59256afb88295, State: Initialized, Role: FOLLOWER
I20260812 07:06:00.566781 19775 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb4472c41fcc4e2ca6f59256afb88295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb4472c41fcc4e2ca6f59256afb88295" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 36111 } }
I20260812 07:06:00.567122 19775 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb4472c41fcc4e2ca6f59256afb88295 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:06:00.567420 19775 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb4472c41fcc4e2ca6f59256afb88295 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:06:00.567690 19775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb4472c41fcc4e2ca6f59256afb88295 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:00.571045 19775 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb4472c41fcc4e2ca6f59256afb88295 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb4472c41fcc4e2ca6f59256afb88295" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 36111 } }
I20260812 07:06:00.571874 19775 leader_election.cc:302] T 00000000000000000000000000000000 P eb4472c41fcc4e2ca6f59256afb88295 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb4472c41fcc4e2ca6f59256afb88295; no voters:
I20260812 07:06:00.573053 19775 leader_election.cc:288] T 00000000000000000000000000000000 P eb4472c41fcc4e2ca6f59256afb88295 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:06:00.573339 19780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb4472c41fcc4e2ca6f59256afb88295 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:06:00.575142 19780 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb4472c41fcc4e2ca6f59256afb88295 [term 1 LEADER]: Becoming Leader. State: Replica: eb4472c41fcc4e2ca6f59256afb88295, State: Running, Role: LEADER
I20260812 07:06:00.575913 19780 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb4472c41fcc4e2ca6f59256afb88295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb4472c41fcc4e2ca6f59256afb88295" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 36111 } }
I20260812 07:06:00.576594 19775 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb4472c41fcc4e2ca6f59256afb88295 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:06:00.580325 19782 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb4472c41fcc4e2ca6f59256afb88295 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb4472c41fcc4e2ca6f59256afb88295" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb4472c41fcc4e2ca6f59256afb88295" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 36111 } } }
I20260812 07:06:00.580852 19783 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb4472c41fcc4e2ca6f59256afb88295 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb4472c41fcc4e2ca6f59256afb88295. Latest consensus state: current_term: 1 leader_uuid: "eb4472c41fcc4e2ca6f59256afb88295" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb4472c41fcc4e2ca6f59256afb88295" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 36111 } } }
I20260812 07:06:00.581519 19782 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb4472c41fcc4e2ca6f59256afb88295 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:00.583299 19783 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb4472c41fcc4e2ca6f59256afb88295 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:00.586618 19789 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:06:00.597954 19789 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:06:00.599038 19775 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 07:06:00.625993 19789 catalog_manager.cc:1347] Generated new cluster ID: 873329d17db840c980010bbb741a008e
I20260812 07:06:00.626240 19789 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:06:00.672708 19789 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:06:00.675879 19789 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:06:00.693010 19789 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb4472c41fcc4e2ca6f59256afb88295: Generated new TSK 0
I20260812 07:06:00.694504 19789 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
I20260812 07:06:01.546794 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.61.20:51595/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.1:0
--local_ip_for_outbound_sockets=127.6.61.1
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:36111
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:43813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 07:06:01 dist-test-slave-69rg krb5kdc[19154](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786518361, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260812 07:06:02.284911 19827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:02.285650 19827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:02.286002 19827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:02.305235 19827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:02.305521 19827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:02.305910 19827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.1
I20260812 07:06:02.329561 19827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:43813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.61.20:51595/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:36111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.1
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:06:02.331326 19827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:02.333174 19827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:02.356976 19840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:02.358532 19843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:02.359288 19841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:02.360960 19827 server_base.cc:1034] running on GCE node
I20260812 07:06:02.362047 19827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:02.363656 19827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:02.365003 19827 hybrid_clock.cc:648] HybridClock initialized: now 1786518362364832 us; error 108 us; skew 500 ppm
I20260812 07:06:02.365494 19827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:06:02 dist-test-slave-69rg krb5kdc[19154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518362, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:02.370296 19827 init.cc:377] Logged in from keytab as kudu/127.6.61.1@KRBTEST.COM (short username kudu)
I20260812 07:06:02.372997 19827 webserver.cc:460] Webserver started at http://127.6.61.1:45635/ using document root <none> and password file <none>
I20260812 07:06:02.374004 19827 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:02.374226 19827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:02.374701 19827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 07:06:02 dist-test-slave-69rg krb5kdc[19154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518362, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for HTTP/127.6.61.20@KRBTEST.COM
I20260812 07:06:02.411477 19827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-0/data/instance:
uuid: "413c933d6f854de5bd18649561122ae8"
format_stamp: "Formatted at 2026-08-12 07:06:02 on dist-test-slave-69rg"
server_key: "63db13603b2ceb783538cf7fffad4c93"
server_key_iv: "738a60b9aca1e0daf7fa1e199e85edb1"
server_key_version: "kuduclusterkey@0"
I20260812 07:06:02.413197 19827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance:
uuid: "413c933d6f854de5bd18649561122ae8"
format_stamp: "Formatted at 2026-08-12 07:06:02 on dist-test-slave-69rg"
server_key: "63db13603b2ceb783538cf7fffad4c93"
server_key_iv: "738a60b9aca1e0daf7fa1e199e85edb1"
server_key_version: "kuduclusterkey@0"
I20260812 07:06:02.428905 19827 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.005s sys 0.007s
I20260812 07:06:02.456002 19850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:02.458294 19827 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260812 07:06:02.458588 19827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-0/wal
uuid: "413c933d6f854de5bd18649561122ae8"
format_stamp: "Formatted at 2026-08-12 07:06:02 on dist-test-slave-69rg"
server_key: "63db13603b2ceb783538cf7fffad4c93"
server_key_iv: "738a60b9aca1e0daf7fa1e199e85edb1"
server_key_version: "kuduclusterkey@0"
I20260812 07:06:02.459023 19827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:02.569294 19827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:02.577135 19827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:02.577868 19827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:02.586238 19827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:06:02.595832 19827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:06:02.596092 19827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:02.596431 19827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:06:02.596622 19827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 07:06:02 dist-test-slave-69rg krb5kdc[19154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518361, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for HTTP/127.6.61.20@KRBTEST.COM
I20260812 07:06:02.973845 19827 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.1:44977
I20260812 07:06:02.974031 19984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.1:44977 every 8 connection(s)
I20260812 07:06:02.976465 19827 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
I20260812 07:06:02.982671 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 19827
I20260812 07:06:02.982941 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance
I20260812 07:06:02.984062 6388 external_mini_cluster.cc:1428] Setting key 49f1394a1106c1521f12e555d58766b9
Aug 12 07:06:03 dist-test-slave-69rg krb5kdc[19154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518362, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.61.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.2@KRBTEST.COM" created.
I20260812 07:06:03.052284 19988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:02.984984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:59385 (local address 127.6.61.62:36111)
0812 07:06:03.004154 (+ 19170us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:03.004163 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:03.019190 (+ 15027us) server_negotiation.cc:407] Connection header received
0812 07:06:03.022279 (+ 3089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:03.022286 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:03.022433 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:03.022557 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:03.028820 (+ 6263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:03.030414 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:03.033645 (+ 3231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:03.034697 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:03.034825 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:03.042621 (+ 7796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:03.042633 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:03.042640 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:03.042690 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:03.047054 (+ 4364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:03.048419 (+ 1365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:03.048427 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:03.048430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:03.048668 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:03.049414 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:03.049423 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:03.049429 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:03.050018 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:03.050169 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:03.051373 (+ 1204us) server_negotiation.cc:299] Negotiation successful
0812 07:06:03.051677 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18912,"thread_start_us":19133,"threads_started":1}
I20260812 07:06:03.055135 19987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:02.981382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:36111 (local address 127.6.61.1:59385)
0812 07:06:03.010785 (+ 29403us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:03.010858 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:03.021952 (+ 11094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:03.026280 (+ 4328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:03.026296 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:03.027288 (+ 992us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:03.028578 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:03.028613 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:03.030661 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:03.030668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:03.033462 (+ 2794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:03.033482 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:03.035381 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:03.035393 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:03.035815 (+ 422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:03.036991 (+ 1176us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:06:03.037052 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:03.042167 (+ 5115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:03.047395 (+ 5228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:03.047414 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:03.047433 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:03.048243 (+ 810us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:03.048885 (+ 642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:03.048892 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:03.048898 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:03.049268 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:03.050286 (+ 1018us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:03.050298 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:03.050857 (+ 559us) client_negotiation.cc:769] Sending connection context
0812 07:06:03.052578 (+ 1721us) client_negotiation.cc:240] Negotiation successful
0812 07:06:03.053272 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28841,"thread_start_us":32305,"threads_started":1}
I20260812 07:06:03.057750 19985 heartbeater.cc:347] Connected to a master server at 127.6.61.62:36111
I20260812 07:06:03.058159 19985 heartbeater.cc:464] Registering TS with master...
I20260812 07:06:03.059428 19985 heartbeater.cc:511] Master 127.6.61.62:36111 requested a full tablet report, sending...
I20260812 07:06:03.062572 19733 ts_manager.cc:194] Registered new tserver with Master: 413c933d6f854de5bd18649561122ae8 (127.6.61.1:44977)
I20260812 07:06:03.065038 19733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.1@KRBTEST.COM'} at 127.6.61.1:59385
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
I20260812 07:06:03.093861 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.61.20:51595/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.2:0
--local_ip_for_outbound_sockets=127.6.61.2
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:36111
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:43813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 07:06:03 dist-test-slave-69rg krb5kdc[19154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518361, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for HTTP/127.6.61.20@KRBTEST.COM
W20260812 07:06:03.835729 19991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:03.836038 19991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:03.836328 19991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:03.845934 19991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:03.846107 19991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:03.846361 19991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.2
I20260812 07:06:03.859283 19991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:43813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.61.20:51595/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:36111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.2
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:06:03.860872 19991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:03.862444 19991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:03.882280 20007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:03.883358 20006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:03.883689 20010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:03.885334 19991 server_base.cc:1034] running on GCE node
I20260812 07:06:03.886248 19991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:03.887835 19991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:03.889168 19991 hybrid_clock.cc:648] HybridClock initialized: now 1786518363889008 us; error 100 us; skew 500 ppm
I20260812 07:06:03.889632 19991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:06:03 dist-test-slave-69rg krb5kdc[19154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518363, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:03.894603 19991 init.cc:377] Logged in from keytab as kudu/127.6.61.2@KRBTEST.COM (short username kudu)
I20260812 07:06:03.897440 19991 webserver.cc:460] Webserver started at http://127.6.61.2:40851/ using document root <none> and password file <none>
I20260812 07:06:03.898320 19991 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:03.898530 19991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:03.898996 19991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 07:06:03 dist-test-slave-69rg krb5kdc[19154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518363, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for HTTP/127.6.61.20@KRBTEST.COM
I20260812 07:06:03.916234 19991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-1/data/instance:
uuid: "5fb77b13af044e2fa7eee8ffab1a3f43"
format_stamp: "Formatted at 2026-08-12 07:06:03 on dist-test-slave-69rg"
server_key: "e48f16511ffb03d13a04f7c0afe9a645"
server_key_iv: "db3f266f2abc29124b16a712e5897bf8"
server_key_version: "kuduclusterkey@0"
I20260812 07:06:03.917390 19991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance:
uuid: "5fb77b13af044e2fa7eee8ffab1a3f43"
format_stamp: "Formatted at 2026-08-12 07:06:03 on dist-test-slave-69rg"
server_key: "e48f16511ffb03d13a04f7c0afe9a645"
server_key_iv: "db3f266f2abc29124b16a712e5897bf8"
server_key_version: "kuduclusterkey@0"
I20260812 07:06:03.924746 19991 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20260812 07:06:03.944677 20017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:03.946575 19991 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 07:06:03.946815 19991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-1/wal
uuid: "5fb77b13af044e2fa7eee8ffab1a3f43"
format_stamp: "Formatted at 2026-08-12 07:06:03 on dist-test-slave-69rg"
server_key: "e48f16511ffb03d13a04f7c0afe9a645"
server_key_iv: "db3f266f2abc29124b16a712e5897bf8"
server_key_version: "kuduclusterkey@0"
I20260812 07:06:03.947167 19991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:03.986927 19991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:03.993100 19991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:03.993558 19991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:03.995298 19991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:06:03.997691 19991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:06:03.997859 19991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:03.998102 19991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:06:03.998220 19991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:04.069736 19985 heartbeater.cc:503] Master 127.6.61.62:36111 was elected leader, sending a full tablet report...
I20260812 07:06:04.111807 19991 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.2:45871
I20260812 07:06:04.111898 20146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.2:45871 every 8 connection(s)
I20260812 07:06:04.113576 19991 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
I20260812 07:06:04.120328 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 19991
I20260812 07:06:04.120528 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance
I20260812 07:06:04.121172 6388 external_mini_cluster.cc:1428] Setting key cea53c7b35d129fb102eddea85c38c6f
Aug 12 07:06:04 dist-test-slave-69rg krb5kdc[19154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518363, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:06:04.146482 20150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:04.117971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:51929 (local address 127.6.61.62:36111)
0812 07:06:04.118953 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:04.118960 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:04.121072 (+ 2112us) server_negotiation.cc:407] Connection header received
0812 07:06:04.124417 (+ 3345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:04.124423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:04.124502 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:04.124620 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:04.126755 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:04.127755 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:04.129890 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:04.130473 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:04.130622 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:04.136057 (+ 5435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:04.136063 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:04.136067 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:04.136147 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:04.141565 (+ 5418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:04.142846 (+ 1281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:04.142852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:04.142856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:04.143039 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:04.143743 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:04.143748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:04.143752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:04.144229 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:04.144381 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:04.145519 (+ 1138us) server_negotiation.cc:299] Negotiation successful
0812 07:06:04.145805 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":813,"thread_start_us":643,"threads_started":1}
I20260812 07:06:04.148154 20151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:04.118844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:36111 (local address 127.6.61.2:51929)
0812 07:06:04.120652 (+ 1808us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:04.120701 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:04.124048 (+ 3347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:04.124954 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:04.124964 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:04.125668 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:04.126568 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:04.126588 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:04.127921 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:04.127926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:04.129686 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:04.129716 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:04.130832 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:04.130842 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:04.131147 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:04.132080 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:06:04.132104 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:04.135711 (+ 3607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:04.141753 (+ 6042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:04.141768 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:04.141791 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:04.142683 (+ 892us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:04.143193 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:04.143201 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:04.143208 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:04.143594 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:04.144442 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:04.144455 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:04.144957 (+ 502us) client_negotiation.cc:769] Sending connection context
0812 07:06:04.145346 (+ 389us) client_negotiation.cc:240] Negotiation successful
0812 07:06:04.146179 (+ 833us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1418,"thread_start_us":1011,"threads_started":1}
I20260812 07:06:04.151412 20147 heartbeater.cc:347] Connected to a master server at 127.6.61.62:36111
I20260812 07:06:04.152038 20147 heartbeater.cc:464] Registering TS with master...
I20260812 07:06:04.153785 20147 heartbeater.cc:511] Master 127.6.61.62:36111 requested a full tablet report, sending...
I20260812 07:06:04.157081 19733 ts_manager.cc:194] Registered new tserver with Master: 5fb77b13af044e2fa7eee8ffab1a3f43 (127.6.61.2:45871)
I20260812 07:06:04.159044 19733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.2@KRBTEST.COM'} at 127.6.61.2:51929
WARNING: no policy specified for kudu/127.6.61.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
I20260812 07:06:04.206172 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.61.20:51595/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.3:0
--local_ip_for_outbound_sockets=127.6.61.3
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:36111
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:43813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:06:04.811195 20155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:04.811628 20155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:04.811861 20155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:04.821537 20155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:04.821768 20155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:04.822016 20155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.3
I20260812 07:06:04.835758 20155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:43813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.61.20:51595/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:36111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.3
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:06:04.837373 20155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:04.839259 20155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:04.858960 20161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:04.860149 20165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:04.860176 20163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:04.861685 20155 server_base.cc:1034] running on GCE node
I20260812 07:06:04.862444 20155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:04.863612 20155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:04.864966 20155 hybrid_clock.cc:648] HybridClock initialized: now 1786518364864826 us; error 59 us; skew 500 ppm
I20260812 07:06:04.865516 20155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:06:04 dist-test-slave-69rg krb5kdc[19154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518364, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:04.870131 20155 init.cc:377] Logged in from keytab as kudu/127.6.61.3@KRBTEST.COM (short username kudu)
I20260812 07:06:04.872008 20155 webserver.cc:460] Webserver started at http://127.6.61.3:34913/ using document root <none> and password file <none>
I20260812 07:06:04.872887 20155 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:04.873051 20155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:04.873430 20155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 07:06:04 dist-test-slave-69rg krb5kdc[19154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518364, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for HTTP/127.6.61.20@KRBTEST.COM
I20260812 07:06:04.889019 20155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-2/data/instance:
uuid: "cbdf58ca4b9b4bc9a8241952bb686067"
format_stamp: "Formatted at 2026-08-12 07:06:04 on dist-test-slave-69rg"
server_key: "8f355129cc8bd46f8852b1306ec21193"
server_key_iv: "ef0b02ade4ed29538d443da35d05e28e"
server_key_version: "kuduclusterkey@0"
I20260812 07:06:04.890324 20155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance:
uuid: "cbdf58ca4b9b4bc9a8241952bb686067"
format_stamp: "Formatted at 2026-08-12 07:06:04 on dist-test-slave-69rg"
server_key: "8f355129cc8bd46f8852b1306ec21193"
server_key_iv: "ef0b02ade4ed29538d443da35d05e28e"
server_key_version: "kuduclusterkey@0"
I20260812 07:06:04.897353 20155 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260812 07:06:04.915654 20173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:04.917531 20155 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.001s
I20260812 07:06:04.917925 20155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-2/wal
uuid: "cbdf58ca4b9b4bc9a8241952bb686067"
format_stamp: "Formatted at 2026-08-12 07:06:04 on dist-test-slave-69rg"
server_key: "8f355129cc8bd46f8852b1306ec21193"
server_key_iv: "ef0b02ade4ed29538d443da35d05e28e"
server_key_version: "kuduclusterkey@0"
I20260812 07:06:04.918293 20155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:04.956243 20155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:04.962109 20155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:04.962607 20155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:04.964169 20155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:06:04.966766 20155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:06:04.966943 20155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:04.967212 20155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:06:04.967357 20155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:05.050369 20155 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.3:44119
I20260812 07:06:05.050446 20309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.3:44119 every 8 connection(s)
I20260812 07:06:05.052052 20155 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
I20260812 07:06:05.056921 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 20155
I20260812 07:06:05.057184 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance
I20260812 07:06:05.057873 6388 external_mini_cluster.cc:1428] Setting key a51f7b03e6a1fe45a2789b1a44e83bb9
Aug 12 07:06:05 dist-test-slave-69rg krb5kdc[19154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518364, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:06:05.079710 20313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:05.055719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:52679 (local address 127.6.61.62:36111)
0812 07:06:05.056710 (+ 991us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:05.056715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:05.058097 (+ 1382us) server_negotiation.cc:407] Connection header received
0812 07:06:05.061017 (+ 2920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:05.061023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:05.061114 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:05.061230 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:05.063284 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:05.064190 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:05.066369 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:05.066973 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:05.067114 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:05.071518 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:05.071524 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:05.071527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:05.071560 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:05.075892 (+ 4332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:05.076738 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:05.076743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:05.076750 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:05.076962 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:05.077510 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:05.077515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:05.077517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:05.077952 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:05.078168 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:05.078911 (+ 743us) server_negotiation.cc:299] Negotiation successful
0812 07:06:05.079182 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":887,"thread_start_us":646,"threads_started":1}
I20260812 07:06:05.080839 20314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:05.056203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:36111 (local address 127.6.61.3:52679)
0812 07:06:05.057892 (+ 1689us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:05.057960 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:05.060718 (+ 2758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:05.061580 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:05.061589 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:05.062306 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:05.063105 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:05.063123 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:05.064351 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:05.064361 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:05.066170 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:05.066183 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:05.067334 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:05.067352 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:05.067696 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:05.068529 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:06:05.068552 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:05.071275 (+ 2723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:05.076041 (+ 4766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:05.076050 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:05.076070 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:05.076600 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:05.077082 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:05.077086 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:05.077090 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:05.077378 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:05.078140 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:05.078148 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:05.078491 (+ 343us) client_negotiation.cc:769] Sending connection context
0812 07:06:05.078815 (+ 324us) client_negotiation.cc:240] Negotiation successful
0812 07:06:05.079483 (+ 668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1310,"thread_start_us":906,"threads_started":1}
I20260812 07:06:05.082640 20310 heartbeater.cc:347] Connected to a master server at 127.6.61.62:36111
I20260812 07:06:05.083032 20310 heartbeater.cc:464] Registering TS with master...
I20260812 07:06:05.083958 20310 heartbeater.cc:511] Master 127.6.61.62:36111 requested a full tablet report, sending...
I20260812 07:06:05.086014 19732 ts_manager.cc:194] Registered new tserver with Master: cbdf58ca4b9b4bc9a8241952bb686067 (127.6.61.3:44119)
I20260812 07:06:05.087396 19732 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.3@KRBTEST.COM'} at 127.6.61.3:52679
I20260812 07:06:05.087600 6388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 07:06:05.117486 20313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:05.094069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39348 (local address 127.6.61.62:36111)
0812 07:06:05.094320 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:05.094327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:05.095147 (+ 820us) server_negotiation.cc:407] Connection header received
0812 07:06:05.095516 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:05.095520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:05.095625 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:05.095709 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:05.097428 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:05.098437 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:05.100591 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:05.101268 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:05.101377 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:05.109101 (+ 7724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:05.109107 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:05.109110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:05.109149 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:05.113847 (+ 4698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:05.114782 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:05.114788 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:05.114792 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:05.115062 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:05.115582 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:05.115587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:05.115591 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:05.116087 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:05.116441 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:05.116766 (+ 325us) server_negotiation.cc:299] Negotiation successful
0812 07:06:05.117028 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 07:06:05.122373 19732 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:06:05.132393 19732 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:06:05.164108 20147 heartbeater.cc:503] Master 127.6.61.62:36111 was elected leader, sending a full tablet report...
I20260812 07:06:05.178508 20329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:05.149265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37806 (local address 127.6.61.2:45871)
0812 07:06:05.162444 (+ 13179us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:05.162459 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:05.162947 (+ 488us) server_negotiation.cc:407] Connection header received
0812 07:06:05.163067 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:05.163098 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:05.163511 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:05.163947 (+ 436us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:05.166796 (+ 2849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:05.169428 (+ 2632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:05.174035 (+ 4607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:05.175828 (+ 1793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:05.175966 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:05.176315 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:05.177671 (+ 1356us) server_negotiation.cc:299] Negotiation successful
0812 07:06:05.177934 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12872,"thread_start_us":12626,"threads_started":1}
I20260812 07:06:05.179051 20327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:05.148442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54970 (local address 127.6.61.1:44977)
0812 07:06:05.164013 (+ 15571us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:05.164020 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:05.164042 (+ 22us) server_negotiation.cc:407] Connection header received
0812 07:06:05.164116 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:05.164120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:05.164399 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:05.164614 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:05.169277 (+ 4663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:05.171243 (+ 1966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:05.175076 (+ 3833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:05.177204 (+ 2128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:05.177603 (+ 399us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:05.177964 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:05.178252 (+ 288us) server_negotiation.cc:299] Negotiation successful
0812 07:06:05.178464 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15382,"thread_start_us":15143,"threads_started":1}
I20260812 07:06:05.179128 20328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:05.148195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:44977 (local address 127.0.0.1:54970)
0812 07:06:05.160657 (+ 12462us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:05.160737 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:05.161043 (+ 306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:05.164645 (+ 3602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:05.164649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:05.164676 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:05.166016 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:05.166032 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:05.171456 (+ 5424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:05.171465 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:05.174863 (+ 3398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:05.174880 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:05.177680 (+ 2800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:05.177689 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:05.177985 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:05.178027 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 07:06:05.178191 (+ 164us) client_negotiation.cc:240] Negotiation successful
0812 07:06:05.178361 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12003,"mutex_wait_us":25,"thread_start_us":14774,"threads_started":1}
I20260812 07:06:05.179778 20325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:05.148715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:45871 (local address 127.0.0.1:37806)
0812 07:06:05.162626 (+ 13911us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:05.162647 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:05.162804 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:05.163790 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:05.163798 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:05.163842 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:05.165632 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:05.165647 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:05.169724 (+ 4077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:05.169738 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:05.173806 (+ 4068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:05.173852 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:05.176169 (+ 2317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:05.176177 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:05.176443 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:05.176520 (+ 77us) client_negotiation.cc:769] Sending connection context
0812 07:06:05.176679 (+ 159us) client_negotiation.cc:240] Negotiation successful
0812 07:06:05.176941 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13767,"mutex_wait_us":20,"thread_start_us":13794,"threads_started":1}
I20260812 07:06:05.182269 20330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:05.166020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:44119 (local address 127.0.0.1:55186)
0812 07:06:05.167225 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:05.167264 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:05.167412 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:05.169803 (+ 2391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:05.169808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:05.169858 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:05.172774 (+ 2916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:05.172787 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:05.174672 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:05.174678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:05.178437 (+ 3759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:05.178461 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:05.181255 (+ 2794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:05.181260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:05.181533 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:05.181566 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 07:06:05.181647 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 07:06:05.181814 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1028,"mutex_wait_us":19,"thread_start_us":824,"threads_started":1}
I20260812 07:06:05.182926 20331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:05.166709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55186 (local address 127.6.61.3:44119)
0812 07:06:05.169289 (+ 2580us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:05.169295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:05.169333 (+ 38us) server_negotiation.cc:407] Connection header received
0812 07:06:05.169405 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:05.169410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:05.169594 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:05.169806 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:05.173026 (+ 3220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:05.174459 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:05.178665 (+ 4206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:05.180604 (+ 1939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:05.180741 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:05.181046 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:05.182559 (+ 1513us) server_negotiation.cc:299] Negotiation successful
0812 07:06:05.182683 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2402,"thread_start_us":2192,"threads_started":1}
I20260812 07:06:05.184278 20075 tablet_service.cc:1467] Processing CreateTablet for tablet 2979871991d641b98dc2f89d47539ef4 (DEFAULT_TABLE table=test-table [id=90ee62f4cee145e2888ee4af66334ccf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:06:05.184321 19909 tablet_service.cc:1467] Processing CreateTablet for tablet 2979871991d641b98dc2f89d47539ef4 (DEFAULT_TABLE table=test-table [id=90ee62f4cee145e2888ee4af66334ccf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:06:05.186359 19909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2979871991d641b98dc2f89d47539ef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:05.186359 20075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2979871991d641b98dc2f89d47539ef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:05.186336 20237 tablet_service.cc:1467] Processing CreateTablet for tablet 2979871991d641b98dc2f89d47539ef4 (DEFAULT_TABLE table=test-table [id=90ee62f4cee145e2888ee4af66334ccf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:06:05.187851 20237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2979871991d641b98dc2f89d47539ef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:05.203217 20332 tablet_bootstrap.cc:492] T 2979871991d641b98dc2f89d47539ef4 P cbdf58ca4b9b4bc9a8241952bb686067: Bootstrap starting.
I20260812 07:06:05.208431 20333 tablet_bootstrap.cc:492] T 2979871991d641b98dc2f89d47539ef4 P 413c933d6f854de5bd18649561122ae8: Bootstrap starting.
I20260812 07:06:05.210496 20332 tablet_bootstrap.cc:654] T 2979871991d641b98dc2f89d47539ef4 P cbdf58ca4b9b4bc9a8241952bb686067: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:05.211030 20334 tablet_bootstrap.cc:492] T 2979871991d641b98dc2f89d47539ef4 P 5fb77b13af044e2fa7eee8ffab1a3f43: Bootstrap starting.
I20260812 07:06:05.213267 20332 log.cc:824] T 2979871991d641b98dc2f89d47539ef4 P cbdf58ca4b9b4bc9a8241952bb686067: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:05.215000 20333 tablet_bootstrap.cc:654] T 2979871991d641b98dc2f89d47539ef4 P 413c933d6f854de5bd18649561122ae8: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:05.215909 20334 tablet_bootstrap.cc:654] T 2979871991d641b98dc2f89d47539ef4 P 5fb77b13af044e2fa7eee8ffab1a3f43: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:05.216498 20333 log.cc:824] T 2979871991d641b98dc2f89d47539ef4 P 413c933d6f854de5bd18649561122ae8: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:05.217654 20334 log.cc:824] T 2979871991d641b98dc2f89d47539ef4 P 5fb77b13af044e2fa7eee8ffab1a3f43: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:05.218289 20332 tablet_bootstrap.cc:492] T 2979871991d641b98dc2f89d47539ef4 P cbdf58ca4b9b4bc9a8241952bb686067: No bootstrap required, opened a new log
I20260812 07:06:05.218700 20332 ts_tablet_manager.cc:1397] T 2979871991d641b98dc2f89d47539ef4 P cbdf58ca4b9b4bc9a8241952bb686067: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.002s
I20260812 07:06:05.220427 20333 tablet_bootstrap.cc:492] T 2979871991d641b98dc2f89d47539ef4 P 413c933d6f854de5bd18649561122ae8: No bootstrap required, opened a new log
I20260812 07:06:05.220733 20333 ts_tablet_manager.cc:1397] T 2979871991d641b98dc2f89d47539ef4 P 413c933d6f854de5bd18649561122ae8: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260812 07:06:05.221997 20334 tablet_bootstrap.cc:492] T 2979871991d641b98dc2f89d47539ef4 P 5fb77b13af044e2fa7eee8ffab1a3f43: No bootstrap required, opened a new log
I20260812 07:06:05.222319 20334 ts_tablet_manager.cc:1397] T 2979871991d641b98dc2f89d47539ef4 P 5fb77b13af044e2fa7eee8ffab1a3f43: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260812 07:06:05.230629 20332 raft_consensus.cc:348] T 2979871991d641b98dc2f89d47539ef4 P cbdf58ca4b9b4bc9a8241952bb686067 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413c933d6f854de5bd18649561122ae8" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 44977 } } peers { permanent_uuid: "cbdf58ca4b9b4bc9a8241952bb686067" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 44119 } } peers { permanent_uuid: "5fb77b13af044e2fa7eee8ffab1a3f43" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45871 } }
I20260812 07:06:05.231077 20332 raft_consensus.cc:374] T 2979871991d641b98dc2f89d47539ef4 P cbdf58ca4b9b4bc9a8241952bb686067 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:05.231267 20332 raft_consensus.cc:729] T 2979871991d641b98dc2f89d47539ef4 P cbdf58ca4b9b4bc9a8241952bb686067 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbdf58ca4b9b4bc9a8241952bb686067, State: Initialized, Role: FOLLOWER
I20260812 07:06:05.232034 20332 consensus_queue.cc:260] T 2979871991d641b98dc2f89d47539ef4 P cbdf58ca4b9b4bc9a8241952bb686067 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413c933d6f854de5bd18649561122ae8" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 44977 } } peers { permanent_uuid: "cbdf58ca4b9b4bc9a8241952bb686067" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 44119 } } peers { permanent_uuid: "5fb77b13af044e2fa7eee8ffab1a3f43" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45871 } }
I20260812 07:06:05.233578 20310 heartbeater.cc:503] Master 127.6.61.62:36111 was elected leader, sending a full tablet report...
I20260812 07:06:05.234579 20332 ts_tablet_manager.cc:1428] T 2979871991d641b98dc2f89d47539ef4 P cbdf58ca4b9b4bc9a8241952bb686067: Time spent starting tablet: real 0.016s user 0.009s sys 0.007s
I20260812 07:06:05.236869 20333 raft_consensus.cc:348] T 2979871991d641b98dc2f89d47539ef4 P 413c933d6f854de5bd18649561122ae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413c933d6f854de5bd18649561122ae8" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 44977 } } peers { permanent_uuid: "cbdf58ca4b9b4bc9a8241952bb686067" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 44119 } } peers { permanent_uuid: "5fb77b13af044e2fa7eee8ffab1a3f43" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45871 } }
I20260812 07:06:05.237504 20333 raft_consensus.cc:374] T 2979871991d641b98dc2f89d47539ef4 P 413c933d6f854de5bd18649561122ae8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:05.237672 20333 raft_consensus.cc:729] T 2979871991d641b98dc2f89d47539ef4 P 413c933d6f854de5bd18649561122ae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 413c933d6f854de5bd18649561122ae8, State: Initialized, Role: FOLLOWER
I20260812 07:06:05.238204 20334 raft_consensus.cc:348] T 2979871991d641b98dc2f89d47539ef4 P 5fb77b13af044e2fa7eee8ffab1a3f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413c933d6f854de5bd18649561122ae8" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 44977 } } peers { permanent_uuid: "cbdf58ca4b9b4bc9a8241952bb686067" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 44119 } } peers { permanent_uuid: "5fb77b13af044e2fa7eee8ffab1a3f43" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45871 } }
I20260812 07:06:05.238510 20333 consensus_queue.cc:260] T 2979871991d641b98dc2f89d47539ef4 P 413c933d6f854de5bd18649561122ae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413c933d6f854de5bd18649561122ae8" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 44977 } } peers { permanent_uuid: "cbdf58ca4b9b4bc9a8241952bb686067" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 44119 } } peers { permanent_uuid: "5fb77b13af044e2fa7eee8ffab1a3f43" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45871 } }
I20260812 07:06:05.238857 20334 raft_consensus.cc:374] T 2979871991d641b98dc2f89d47539ef4 P 5fb77b13af044e2fa7eee8ffab1a3f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:05.239044 20334 raft_consensus.cc:729] T 2979871991d641b98dc2f89d47539ef4 P 5fb77b13af044e2fa7eee8ffab1a3f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fb77b13af044e2fa7eee8ffab1a3f43, State: Initialized, Role: FOLLOWER
I20260812 07:06:05.239908 20334 consensus_queue.cc:260] T 2979871991d641b98dc2f89d47539ef4 P 5fb77b13af044e2fa7eee8ffab1a3f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413c933d6f854de5bd18649561122ae8" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 44977 } } peers { permanent_uuid: "cbdf58ca4b9b4bc9a8241952bb686067" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 44119 } } peers { permanent_uuid: "5fb77b13af044e2fa7eee8ffab1a3f43" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45871 } }
I20260812 07:06:05.241775 20333 ts_tablet_manager.cc:1428] T 2979871991d641b98dc2f89d47539ef4 P 413c933d6f854de5bd18649561122ae8: Time spent starting tablet: real 0.021s user 0.011s sys 0.011s
I20260812 07:06:05.243409 20334 ts_tablet_manager.cc:1428] T 2979871991d641b98dc2f89d47539ef4 P 5fb77b13af044e2fa7eee8ffab1a3f43: Time spent starting tablet: real 0.021s user 0.012s sys 0.008s
W20260812 07:06:05.273583 19986 tablet.cc:2369] T 2979871991d641b98dc2f89d47539ef4 P 413c933d6f854de5bd18649561122ae8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:06:05.284233 20339 raft_consensus.cc:482] T 2979871991d641b98dc2f89d47539ef4 P 413c933d6f854de5bd18649561122ae8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:06:05.284756 20339 raft_consensus.cc:504] T 2979871991d641b98dc2f89d47539ef4 P 413c933d6f854de5bd18649561122ae8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413c933d6f854de5bd18649561122ae8" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 44977 } } peers { permanent_uuid: "cbdf58ca4b9b4bc9a8241952bb686067" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 44119 } } peers { permanent_uuid: "5fb77b13af044e2fa7eee8ffab1a3f43" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45871 } }
I20260812 07:06:05.287128 20339 leader_election.cc:288] T 2979871991d641b98dc2f89d47539ef4 P 413c933d6f854de5bd18649561122ae8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbdf58ca4b9b4bc9a8241952bb686067 (127.6.61.3:44119), 5fb77b13af044e2fa7eee8ffab1a3f43 (127.6.61.2:45871)
I20260812 07:06:05.300742 20329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:05.288441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:34443 (local address 127.6.61.2:45871)
0812 07:06:05.288766 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:05.288774 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:05.289451 (+ 677us) server_negotiation.cc:407] Connection header received
0812 07:06:05.289863 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:05.289868 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:05.289945 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:05.290055 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:05.291562 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:05.293002 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:05.297628 (+ 4626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:05.299465 (+ 1837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:05.299632 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:05.299808 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:05.300241 (+ 433us) server_negotiation.cc:299] Negotiation successful
0812 07:06:05.300383 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 07:06:05.300763 20331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:05.287919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:56443 (local address 127.6.61.3:44119)
0812 07:06:05.288211 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:05.288229 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:05.290038 (+ 1809us) server_negotiation.cc:407] Connection header received
0812 07:06:05.290364 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:05.290368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:05.290455 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:05.290551 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:05.292025 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:05.293300 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:05.297628 (+ 4328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:05.299213 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:05.299418 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:05.299546 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:05.300184 (+ 638us) server_negotiation.cc:299] Negotiation successful
0812 07:06:05.300318 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260812 07:06:05.301187 20342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:05.287778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:44119 (local address 127.6.61.1:56443)
0812 07:06:05.289923 (+ 2145us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:05.289939 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:05.290163 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:05.290655 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:05.290665 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:05.290698 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:05.291698 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:05.291712 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:05.293564 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:05.293570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:05.297431 (+ 3861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:05.297450 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:05.299527 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:05.299534 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:05.299975 (+ 441us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:05.300011 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 07:06:05.300156 (+ 145us) client_negotiation.cc:240] Negotiation successful
0812 07:06:05.300307 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1944,"mutex_wait_us":65,"thread_start_us":1562,"threads_started":1}
I20260812 07:06:05.301599 20341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:05.287495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:45871 (local address 127.6.61.1:34443)
0812 07:06:05.289293 (+ 1798us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:05.289339 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:05.289607 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:05.290097 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:05.290102 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:05.290203 (+ 101us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:05.291336 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:05.291361 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:05.293183 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:05.293187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:05.297444 (+ 4257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:05.297453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:05.299901 (+ 2448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:05.299910 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:05.300076 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:05.300109 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 07:06:05.300211 (+ 102us) client_negotiation.cc:240] Negotiation successful
0812 07:06:05.300359 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1660,"mutex_wait_us":83,"spinlock_wait_cycles":46336,"thread_start_us":1366,"threads_started":1}
I20260812 07:06:05.302876 20258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2979871991d641b98dc2f89d47539ef4" candidate_uuid: "413c933d6f854de5bd18649561122ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbdf58ca4b9b4bc9a8241952bb686067" is_pre_election: true
I20260812 07:06:05.303109 20096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2979871991d641b98dc2f89d47539ef4" candidate_uuid: "413c933d6f854de5bd18649561122ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fb77b13af044e2fa7eee8ffab1a3f43" is_pre_election: true
I20260812 07:06:05.303627 20258 raft_consensus.cc:2454] T 2979871991d641b98dc2f89d47539ef4 P cbdf58ca4b9b4bc9a8241952bb686067 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 413c933d6f854de5bd18649561122ae8 in term 0.
I20260812 07:06:05.304098 20096 raft_consensus.cc:2454] T 2979871991d641b98dc2f89d47539ef4 P 5fb77b13af044e2fa7eee8ffab1a3f43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 413c933d6f854de5bd18649561122ae8 in term 0.
I20260812 07:06:05.304587 19853 leader_election.cc:302] T 2979871991d641b98dc2f89d47539ef4 P 413c933d6f854de5bd18649561122ae8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 413c933d6f854de5bd18649561122ae8, cbdf58ca4b9b4bc9a8241952bb686067; no voters:
W20260812 07:06:05.305249 20312 tablet.cc:2369] T 2979871991d641b98dc2f89d47539ef4 P cbdf58ca4b9b4bc9a8241952bb686067: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:06:05.305253 20339 raft_consensus.cc:2781] T 2979871991d641b98dc2f89d47539ef4 P 413c933d6f854de5bd18649561122ae8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:06:05.305505 20339 raft_consensus.cc:482] T 2979871991d641b98dc2f89d47539ef4 P 413c933d6f854de5bd18649561122ae8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:06:05.305692 20339 raft_consensus.cc:3037] T 2979871991d641b98dc2f89d47539ef4 P 413c933d6f854de5bd18649561122ae8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:05.309018 20339 raft_consensus.cc:504] T 2979871991d641b98dc2f89d47539ef4 P 413c933d6f854de5bd18649561122ae8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413c933d6f854de5bd18649561122ae8" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 44977 } } peers { permanent_uuid: "cbdf58ca4b9b4bc9a8241952bb686067" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 44119 } } peers { permanent_uuid: "5fb77b13af044e2fa7eee8ffab1a3f43" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45871 } }
I20260812 07:06:05.309965 20339 leader_election.cc:288] T 2979871991d641b98dc2f89d47539ef4 P 413c933d6f854de5bd18649561122ae8 [CANDIDATE]: Term 1 election: Requested vote from peers cbdf58ca4b9b4bc9a8241952bb686067 (127.6.61.3:44119), 5fb77b13af044e2fa7eee8ffab1a3f43 (127.6.61.2:45871)
I20260812 07:06:05.310590 20258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2979871991d641b98dc2f89d47539ef4" candidate_uuid: "413c933d6f854de5bd18649561122ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbdf58ca4b9b4bc9a8241952bb686067"
I20260812 07:06:05.310684 20096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2979871991d641b98dc2f89d47539ef4" candidate_uuid: "413c933d6f854de5bd18649561122ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fb77b13af044e2fa7eee8ffab1a3f43"
I20260812 07:06:05.310869 20258 raft_consensus.cc:3037] T 2979871991d641b98dc2f89d47539ef4 P cbdf58ca4b9b4bc9a8241952bb686067 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:05.311018 20096 raft_consensus.cc:3037] T 2979871991d641b98dc2f89d47539ef4 P 5fb77b13af044e2fa7eee8ffab1a3f43 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:05.314128 20258 raft_consensus.cc:2454] T 2979871991d641b98dc2f89d47539ef4 P cbdf58ca4b9b4bc9a8241952bb686067 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 413c933d6f854de5bd18649561122ae8 in term 1.
I20260812 07:06:05.314213 20096 raft_consensus.cc:2454] T 2979871991d641b98dc2f89d47539ef4 P 5fb77b13af044e2fa7eee8ffab1a3f43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 413c933d6f854de5bd18649561122ae8 in term 1.
I20260812 07:06:05.314769 19853 leader_election.cc:302] T 2979871991d641b98dc2f89d47539ef4 P 413c933d6f854de5bd18649561122ae8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 413c933d6f854de5bd18649561122ae8, cbdf58ca4b9b4bc9a8241952bb686067; no voters:
I20260812 07:06:05.315209 20339 raft_consensus.cc:2781] T 2979871991d641b98dc2f89d47539ef4 P 413c933d6f854de5bd18649561122ae8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:06:05.316035 20339 raft_consensus.cc:686] T 2979871991d641b98dc2f89d47539ef4 P 413c933d6f854de5bd18649561122ae8 [term 1 LEADER]: Becoming Leader. State: Replica: 413c933d6f854de5bd18649561122ae8, State: Running, Role: LEADER
I20260812 07:06:05.316780 20339 consensus_queue.cc:237] T 2979871991d641b98dc2f89d47539ef4 P 413c933d6f854de5bd18649561122ae8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413c933d6f854de5bd18649561122ae8" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 44977 } } peers { permanent_uuid: "cbdf58ca4b9b4bc9a8241952bb686067" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 44119 } } peers { permanent_uuid: "5fb77b13af044e2fa7eee8ffab1a3f43" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45871 } }
I20260812 07:06:05.321961 19732 catalog_manager.cc:5495] T 2979871991d641b98dc2f89d47539ef4 P 413c933d6f854de5bd18649561122ae8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 413c933d6f854de5bd18649561122ae8 (127.6.61.1). New cstate: current_term: 1 leader_uuid: "413c933d6f854de5bd18649561122ae8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413c933d6f854de5bd18649561122ae8" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 44977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbdf58ca4b9b4bc9a8241952bb686067" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 44119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fb77b13af044e2fa7eee8ffab1a3f43" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45871 } health_report { overall_health: UNKNOWN } } }
I20260812 07:06:05.345463 20327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:05.339197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54976 (local address 127.6.61.1:44977)
0812 07:06:05.339438 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:05.339443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:05.339464 (+ 21us) server_negotiation.cc:407] Connection header received
0812 07:06:05.339560 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:05.339564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:05.339643 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:05.339798 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:06:05.340244 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:05.341132 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:05.342990 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:05.343588 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:05.343689 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:05.344309 (+ 620us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:06:05.344890 (+ 581us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:06:05.345005 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:05.345111 (+ 106us) server_negotiation.cc:299] Negotiation successful
0812 07:06:05.345185 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 07:06:05.356980 20096 raft_consensus.cc:1261] T 2979871991d641b98dc2f89d47539ef4 P 5fb77b13af044e2fa7eee8ffab1a3f43 [term 1 FOLLOWER]: Refusing update from remote peer 413c933d6f854de5bd18649561122ae8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:06:05.356980 20258 raft_consensus.cc:1261] T 2979871991d641b98dc2f89d47539ef4 P cbdf58ca4b9b4bc9a8241952bb686067 [term 1 FOLLOWER]: Refusing update from remote peer 413c933d6f854de5bd18649561122ae8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:06:05.358435 20343 consensus_queue.cc:1035] T 2979871991d641b98dc2f89d47539ef4 P 413c933d6f854de5bd18649561122ae8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fb77b13af044e2fa7eee8ffab1a3f43" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 45871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:06:05.358997 20339 consensus_queue.cc:1035] T 2979871991d641b98dc2f89d47539ef4 P 413c933d6f854de5bd18649561122ae8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbdf58ca4b9b4bc9a8241952bb686067" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 44119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 07:06:05.367585 20148 tablet.cc:2369] T 2979871991d641b98dc2f89d47539ef4 P 5fb77b13af044e2fa7eee8ffab1a3f43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:06:05.375057 20347 mvcc.cc:205] Tried to move back new op lower bound from 7317579224487796736 to 7317579224342122496. Current Snapshot: MvccSnapshot[applied={T|T < 7317579224487796736}]
I20260812 07:06:05.394174 19733 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:06:05.394533 19733 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:06:05.399693 19733 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb4472c41fcc4e2ca6f59256afb88295: Sending DeleteTablet for 3 replicas of tablet 2979871991d641b98dc2f89d47539ef4
I20260812 07:06:05.401394 19909 tablet_service.cc:1514] Processing DeleteTablet for tablet 2979871991d641b98dc2f89d47539ef4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:06:05 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:54970
I20260812 07:06:05.401458 20237 tablet_service.cc:1514] Processing DeleteTablet for tablet 2979871991d641b98dc2f89d47539ef4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:06:05 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:55186
I20260812 07:06:05.403317 20075 tablet_service.cc:1514] Processing DeleteTablet for tablet 2979871991d641b98dc2f89d47539ef4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:06:05 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:37806
I20260812 07:06:05.404786 20359 tablet_replica.cc:330] T 2979871991d641b98dc2f89d47539ef4 P 5fb77b13af044e2fa7eee8ffab1a3f43: stopping tablet replica
I20260812 07:06:05.405448 20359 raft_consensus.cc:2229] T 2979871991d641b98dc2f89d47539ef4 P 5fb77b13af044e2fa7eee8ffab1a3f43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:06:05.405735 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 19827
I20260812 07:06:05.406243 20359 raft_consensus.cc:2258] T 2979871991d641b98dc2f89d47539ef4 P 5fb77b13af044e2fa7eee8ffab1a3f43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:06:05.412109 20359 ts_tablet_manager.cc:1905] T 2979871991d641b98dc2f89d47539ef4 P 5fb77b13af044e2fa7eee8ffab1a3f43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:06:05.421087 20357 tablet_replica.cc:330] T 2979871991d641b98dc2f89d47539ef4 P 413c933d6f854de5bd18649561122ae8: stopping tablet replica
I20260812 07:06:05.421751 20358 tablet_replica.cc:330] T 2979871991d641b98dc2f89d47539ef4 P cbdf58ca4b9b4bc9a8241952bb686067: stopping tablet replica
I20260812 07:06:05.421716 20357 raft_consensus.cc:2229] T 2979871991d641b98dc2f89d47539ef4 P 413c933d6f854de5bd18649561122ae8 [term 1 LEADER]: Raft consensus shutting down.
Aug 12 07:06:05 dist-test-slave-69rg krb5kdc[19154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.1@KRBTEST.COM
I20260812 07:06:05.422356 20358 raft_consensus.cc:2229] T 2979871991d641b98dc2f89d47539ef4 P cbdf58ca4b9b4bc9a8241952bb686067 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:06:05.422838 20357 raft_consensus.cc:2258] T 2979871991d641b98dc2f89d47539ef4 P 413c933d6f854de5bd18649561122ae8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:06:05.423516 20358 raft_consensus.cc:2258] T 2979871991d641b98dc2f89d47539ef4 P cbdf58ca4b9b4bc9a8241952bb686067 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:06:05.424551 20359 ts_tablet_manager.cc:1918] T 2979871991d641b98dc2f89d47539ef4 P 5fb77b13af044e2fa7eee8ffab1a3f43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:06:05.425917 20358 ts_tablet_manager.cc:1905] T 2979871991d641b98dc2f89d47539ef4 P cbdf58ca4b9b4bc9a8241952bb686067: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:06:05.425971 20359 log.cc:1196] T 2979871991d641b98dc2f89d47539ef4 P 5fb77b13af044e2fa7eee8ffab1a3f43: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-1/wal/wals/2979871991d641b98dc2f89d47539ef4
I20260812 07:06:05.426689 20359 ts_tablet_manager.cc:1939] T 2979871991d641b98dc2f89d47539ef4 P 5fb77b13af044e2fa7eee8ffab1a3f43: Deleting consensus metadata
I20260812 07:06:05.428689 19714 catalog_manager.cc:4841] TS 5fb77b13af044e2fa7eee8ffab1a3f43 (127.6.61.2:45871): tablet 2979871991d641b98dc2f89d47539ef4 (table test-table [id=90ee62f4cee145e2888ee4af66334ccf]) successfully deleted
I20260812 07:06:05.429245 20357 ts_tablet_manager.cc:1905] T 2979871991d641b98dc2f89d47539ef4 P 413c933d6f854de5bd18649561122ae8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:06:05.433629 20358 ts_tablet_manager.cc:1918] T 2979871991d641b98dc2f89d47539ef4 P cbdf58ca4b9b4bc9a8241952bb686067: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:06:05.433831 20358 log.cc:1196] T 2979871991d641b98dc2f89d47539ef4 P cbdf58ca4b9b4bc9a8241952bb686067: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518208686283-6388-0/minicluster-data/ts-2/wal/wals/2979871991d641b98dc2f89d47539ef4
I20260812 07:06:05.434417 20358 ts_tablet_manager.cc:1939] T 2979871991d641b98dc2f89d47539ef4 P cbdf58ca4b9b4bc9a8241952bb686067: Deleting consensus metadata
I20260812 07:06:05.435272 20327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:05.407502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54984 (local address 127.6.61.1:44977)
0812 07:06:05.407680 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:05.407687 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:05.407707 (+ 20us) server_negotiation.cc:407] Connection header received
0812 07:06:05.407777 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:05.407782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:05.407884 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:05.408001 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:05.410379 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:05.412034 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:05.414993 (+ 2959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:05.416079 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:05.416234 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:05.424387 (+ 8153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:05.424407 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:05.424428 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:05.424503 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:05.430479 (+ 5976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:05.431902 (+ 1423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:05.431912 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:05.431917 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:05.432246 (+ 329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:05.432953 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:05.432958 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:05.432961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:05.433575 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:05.434176 (+ 601us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:05.434373 (+ 197us) server_negotiation.cc:299] Negotiation successful
0812 07:06:05.434722 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 07:06:05.436246 19979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:06:05.438797 19715 catalog_manager.cc:4841] TS cbdf58ca4b9b4bc9a8241952bb686067 (127.6.61.3:44119): tablet 2979871991d641b98dc2f89d47539ef4 (table test-table [id=90ee62f4cee145e2888ee4af66334ccf]) successfully deleted
I20260812 07:06:05.538672 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 19827
I20260812 07:06:05.563161 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 19991
W20260812 07:06:05.563114 19715 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.61.1:44977 (error 108)
I20260812 07:06:05.565384 20330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:05.564769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:44977 (local address 127.0.0.1:54994)
0812 07:06:05.565017 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:05.565170 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260812 07:06:05.565694 19715 catalog_manager.cc:4584] TS 413c933d6f854de5bd18649561122ae8 (127.6.61.1:44977): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2979871991d641b98dc2f89d47539ef4: Network error: Client connection negotiation failed: client connection to 127.6.61.1:44977: connect: Connection refused (error 111)
Aug 12 07:06:05 dist-test-slave-69rg krb5kdc[19154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.2@KRBTEST.COM
I20260812 07:06:05.580214 20329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:05.564058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37814 (local address 127.6.61.2:45871)
0812 07:06:05.564319 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:05.564325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:05.564474 (+ 149us) server_negotiation.cc:407] Connection header received
0812 07:06:05.564737 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:05.564743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:05.564835 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:05.564968 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:05.566856 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:05.567638 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:05.569039 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:05.569474 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:05.569585 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:05.573984 (+ 4399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:05.573999 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:05.574016 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:05.574051 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:05.577167 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:05.577939 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:05.577947 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:05.577949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:05.578131 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:05.578581 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:05.578587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:05.578590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:05.578965 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:05.579296 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:05.579624 (+ 328us) server_negotiation.cc:299] Negotiation successful
0812 07:06:05.579844 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260812 07:06:05.580926 20141 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:06:05.619484 20330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:05.619004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:44977 (local address 127.0.0.1:55006)
0812 07:06:05.619223 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:05.619320 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260812 07:06:05.620632 20330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:05.620242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:44977 (local address 127.0.0.1:55018)
0812 07:06:05.620438 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:05.620522 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260812 07:06:05.672302 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 19991
I20260812 07:06:05.681424 20330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:05.680837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:44977 (local address 127.0.0.1:55028)
0812 07:06:05.681080 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:05.681199 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260812 07:06:05.682641 20330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:05.682184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:44977 (local address 127.0.0.1:55034)
0812 07:06:05.682379 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:05.682478 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260812 07:06:05.695660 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 20155
Aug 12 07:06:05 dist-test-slave-69rg krb5kdc[19154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.3@KRBTEST.COM
I20260812 07:06:05.712361 20331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:05.696530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55198 (local address 127.6.61.3:44119)
0812 07:06:05.696765 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:05.696772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:05.696794 (+ 22us) server_negotiation.cc:407] Connection header received
0812 07:06:05.696934 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:05.696938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:05.697008 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:05.697133 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:05.698564 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:05.699320 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:05.700672 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:05.701074 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:05.701214 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:05.706199 (+ 4985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:05.706217 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:05.706236 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:05.706280 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:05.709212 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:05.710039 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:05.710048 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:05.710050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:05.710202 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:05.710637 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:05.710643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:05.710645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:05.711042 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:05.711374 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:05.711743 (+ 369us) server_negotiation.cc:299] Negotiation successful
0812 07:06:05.711971 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 07:06:05.713127 20303 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:06:05.764111 20330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:05.763495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:44977 (local address 127.0.0.1:55042)
0812 07:06:05.763766 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:05.763879 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260812 07:06:05.765372 20330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:05.764911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:44977 (local address 127.0.0.1:55050)
0812 07:06:05.765131 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:05.765242 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260812 07:06:05.805217 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 20155
I20260812 07:06:05.826597 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 19694
I20260812 07:06:05.827951 19767 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:06:05.943138 20330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:05.942250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:44977 (local address 127.0.0.1:55066)
0812 07:06:05.942523 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:05.942657 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260812 07:06:05.943333 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 19694
I20260812 07:06:05.976667 6388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 07:06:06.322710 6388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 07:06:06.323016 6388 mini_ranger.cc:66] Stopping Ranger...
I20260812 07:06:06.668715 6388 mini_ranger.cc:68] Stopped Ranger
2026-08-12 07:06:06.669 UTC [19188] LOG: received smart shutdown request
2026-08-12 07:06:06.673 UTC [19188] LOG: background worker "logical replication launcher" (PID 19199) exited with exit code 1
2026-08-12 07:06:06.673 UTC [19193] LOG: shutting down
2026-08-12 07:06:06.851 UTC [19188] LOG: database system is shut down
2026-08-12T07:06:06Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (53722 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:06:07 dist-test-slave-69rg krb5kdc[20379](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:06:07 dist-test-slave-69rg krb5kdc[20379](info): set up 2 sockets
Aug 12 07:06:07 dist-test-slave-69rg krb5kdc[20379](info): commencing operation
krb5kdc: starting...
W20260812 07:06:08.238143 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.193s user 0.000s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:06:08 dist-test-slave-69rg krb5kdc[20379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:06:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:06:08Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.61.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.61.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.61.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/rangeradmin_127.6.61.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.61.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/rangeradmin_127.6.61.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.61.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.61.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.61.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/rangerlookup_127.6.61.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.61.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/rangerlookup_127.6.61.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.61.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.61.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.61.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/HTTP_127.6.61.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.61.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/HTTP_127.6.61.20@KRBTEST.COM.keytab.
I20260812 07:06:08.673629 6388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task6k_OM2/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/postgres -l logfile start
2026-08-12 07:06:11.596 UTC [20413] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 07:06:11.596 UTC [20413] LOG: listening on IPv4 address "127.6.61.20", port 36367
2026-08-12 07:06:11.600 UTC [20413] LOG: listening on Unix socket "/tmp/.s.PGSQL.36367"
2026-08-12 07:06:11.625 UTC [20416] LOG: database system was shut down at 2026-08-12 07:06:09 UTC
2026-08-12 07:06:11.630 UTC [20413] LOG: database system is ready to accept connections
I20260812 07:06:12.745445 6388 mini_postgres.cc:96] Postgres bound to 36367
2026-08-12 07:06:12.770 UTC [20426] FATAL: database "slave" does not exist
127.6.61.20:36367 - accepting connections
I20260812 07:06:12.770925 6388 mini_ranger.cc:161] Starting Ranger...
I20260812 07:06:12.805287 6388 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 07:06:13.342470 6388 mini_ranger.cc:87] Created ranger Postgres database
I20260812 07:06:13.342577 6388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ranger-admin
2026-08-12 07:06:13,616 [I] DB FLAVOR :POSTGRES
2026-08-12 07:06:13,616 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 07:06:13,616 [I] Checking connection..
2026-08-12 07:06:13,617 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:36367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 07:06:13,907 [I] Checking connection passed.
2026-08-12 07:06:13,907 [I] --------- Verifying version history table ---------
2026-08-12 07:06:13,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:36367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 07:06:14,178 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 07:06:14,179 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:36367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 07:06:14,452 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 07:06:14,453 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 07:06:14,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:36367/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 07:06:14.716 UTC [20503] WARNING: there is no transaction in progress
2026-08-12 07:06:14,724 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 07:06:14,724 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:36367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 07:06:14,987 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 07:06:14,987 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 07:06:14,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:36367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 07:06:15,234 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:36367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 07:06:15,481 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:36367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-69rg.c.gcp-upstream.internal','N') ;"
2026-08-12 07:06:15,719 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:36367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 07:06:15,986 [I] Table x_portal_user does not exist in database ranger
2026-08-12 07:06:15,986 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:36367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 07:06:16,243 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 07:06:16,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:36367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 07:06:16,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:36367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 07:06:16,758 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 07:06:16,759 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:36367/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 07:06:17.778 UTC [20632] WARNING: there is no transaction in progress
2026-08-12 07:06:17.795 UTC [20632] WARNING: there is no transaction in progress
2026-08-12 07:06:17.813 UTC [20632] WARNING: there is no transaction in progress
2026-08-12 07:06:17.830 UTC [20632] WARNING: there is no transaction in progress
2026-08-12 07:06:17.850 UTC [20632] WARNING: there is no transaction in progress
2026-08-12 07:06:17.873 UTC [20632] WARNING: there is no transaction in progress
2026-08-12 07:06:17.932 UTC [20632] WARNING: there is no transaction in progress
2026-08-12 07:06:17.940 UTC [20632] WARNING: there is no transaction in progress
2026-08-12 07:06:17.957 UTC [20632] WARNING: there is no transaction in progress
2026-08-12 07:06:17.969 UTC [20632] WARNING: there is no transaction in progress
2026-08-12 07:06:17.982 UTC [20632] WARNING: there is no transaction in progress
2026-08-12 07:06:17.996 UTC [20632] WARNING: there is no transaction in progress
2026-08-12 07:06:18.007 UTC [20632] WARNING: there is no transaction in progress
2026-08-12 07:06:18.014 UTC [20632] WARNING: there is no transaction in progress
2026-08-12 07:06:18.022 UTC [20632] WARNING: there is no transaction in progress
2026-08-12 07:06:18.030 UTC [20632] WARNING: there is no transaction in progress
2026-08-12 07:06:18,735 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 07:06:18,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:36367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 07:06:18,986 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 07:06:18,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:36367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 07:06:19,256 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 07:06:19,257 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:36367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 07:06:19,533 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 07:06:19,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:36367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 07:06:19,766 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:36367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 07:06:20,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:36367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-69rg.c.gcp-upstream.internal';"
2026-08-12 07:06:20,301 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 07:06:20,301 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:36367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 07:06:20,595 [I] DB_PATCHES have already been applied
I20260812 07:06:20.601613 6388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ranger-admin:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task6k_OM2/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 07:06:20.601754 6388 mini_ranger.cc:193] Using host: 127.6.61.20
I20260812 07:06:20.624114 6388 mini_ranger.cc:227] Ranger admin URL: http://127.6.61.20:57617
Aug 12, 2026 7:06:20 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 7:06:20 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 7:06:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 7:06:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 7:06:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 7:06:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 7:06:21 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 07:06:21.906203 6388 mini_ranger.cc:160] Time spent starting Ranger: real 9.135s user 0.003s sys 0.060s
Aug 12, 2026 7:06:41 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 7:06:41 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 12, 2026 7:06:41 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 7:06:41 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 7:06:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 7:06:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 7:06:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 7:06:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 7:06:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 7:06:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 7:06:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 7:06:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 7:06:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 7:06:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 7:06:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 7:06:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 7:06:41 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 7:06:42 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 07:06:42.989729 6388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.61.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.61.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.61.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/rangerkms_127.6.61.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.61.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/rangerkms_127.6.61.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.61.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.61.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.61.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/HTTP_127.6.61.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.61.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/HTTP_127.6.61.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 07:06:43 dist-test-slave-69rg krb5kdc[20379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518403, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 07:06:43.222741 6388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 07:06:43.249995 6388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 07:06:44.238451 6388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 07:06:44.238588 6388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ranger-kms
I20260812 07:06:44.238638 6388 mini_ranger_kms.cc:227] Using postgres at 127.6.61.20:36367
2026-08-12 07:06:44,626 [I] DB FLAVOR :POSTGRES
2026-08-12 07:06:44,627 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 07:06:44,627 [I] Checking connection
2026-08-12 07:06:44,627 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:36367/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 07:06:44,896 [I] connection success
2026-08-12 07:06:44,896 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 07:06:44,896 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 07:06:44,897 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:36367/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 07:06:45,144 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 07:06:45,145 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 07:06:45,145 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 07:06:45,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.61.20:36367/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 07:06:45,425 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 07:06:45.918423 6388 mini_ranger_kms.cc:310] Created kms service
I20260812 07:06:46.106103 6388 mini_ranger_kms.cc:326] Created kudu user
I20260812 07:06:46.151973 6388 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 07:06:46 dist-test-slave-69rg krb5kdc[20379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518403, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.61.20@KRBTEST.COM
I20260812 07:06:46.580009 6388 mini_ranger_kms.cc:384] Added ranger policy
I20260812 07:06:46.580381 6388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ranger-kms:/tmp/dist-test-task6k_OM2/build/asan/bin/postgresql.jar:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task6k_OM2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 07:06:46.580447 6388 mini_ranger_kms.cc:240] Using host: 127.6.61.20
I20260812 07:06:46.604420 6388 mini_ranger_kms.cc:276] Ranger KMS PID: 20838
I20260812 07:06:46.604576 6388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.61.20:53803
Aug 12, 2026 7:06:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 7:06:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 7:06:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 7:06:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 7:06:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 7:06:48 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 12 07:06:48 dist-test-slave-69rg krb5kdc[20379](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786518408, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.61.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:49.129153 6388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.906s user 0.044s sys 0.382s
I20260812 07:06:49.129434 6388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 07:06:49.129608 6388 mini_ranger_kms.cc:401] 127.6.61.20:53803/kms/v1/keys
Aug 12 07:06:49 dist-test-slave-69rg krb5kdc[20379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518408, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.61.20@KRBTEST.COM for HTTP/127.6.61.20@KRBTEST.COM
Aug 12 07:06:49 dist-test-slave-69rg krb5kdc[20379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518408, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.61.20@KRBTEST.COM for HTTP/127.6.61.20@KRBTEST.COM
Aug 12 07:06:51 dist-test-slave-69rg krb5kdc[20379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
I20260812 07:06:52.039294 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:39677
--webserver_interface=127.6.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:39677
--ranger_config_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.61.20:53803/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:06:52.406502 20895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:52.406800 20895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:52.406955 20895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:52.414444 20895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:06:52.414554 20895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:52.414652 20895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:52.414744 20895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:06:52.414812 20895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:06:52.425397 20895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.61.20:53803/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:39677
--ranger_config_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:39677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:06:52.426792 20895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:52.428109 20895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:52.445214 20900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:52.445461 20903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:52.445379 20901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:52.446080 20895 server_base.cc:1034] running on GCE node
I20260812 07:06:52.446888 20895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:52.448450 20895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:52.449672 20895 hybrid_clock.cc:648] HybridClock initialized: now 1786518412449631 us; error 39 us; skew 500 ppm
I20260812 07:06:52.450073 20895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:06:52 dist-test-slave-69rg krb5kdc[20379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518412, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:52.453634 20895 init.cc:377] Logged in from keytab as kudu/127.6.61.62@KRBTEST.COM (short username kudu)
I20260812 07:06:52.455482 20895 webserver.cc:460] Webserver started at http://127.6.61.62:37369/ using document root <none> and password file <none>
I20260812 07:06:52.456364 20895 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:52.456507 20895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:52.456912 20895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 07:06:52 dist-test-slave-69rg krb5kdc[20379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518412, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for HTTP/127.6.61.20@KRBTEST.COM
I20260812 07:06:52.515550 20895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "f4b41d60adfc4c0a8bd18700e5e96b52"
format_stamp: "Formatted at 2026-08-12 07:06:52 on dist-test-slave-69rg"
server_key: "5bbd94bffc42c29ea01f497815d7aebb"
server_key_iv: "70f709cb49b3771be7dc0d9bc3b593d6"
server_key_version: "kuduclusterkey@0"
I20260812 07:06:52.516448 20895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "f4b41d60adfc4c0a8bd18700e5e96b52"
format_stamp: "Formatted at 2026-08-12 07:06:52 on dist-test-slave-69rg"
server_key: "5bbd94bffc42c29ea01f497815d7aebb"
server_key_iv: "70f709cb49b3771be7dc0d9bc3b593d6"
server_key_version: "kuduclusterkey@0"
I20260812 07:06:52.522123 20895 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260812 07:06:52.542433 20911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:52.544296 20895 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260812 07:06:52.544519 20895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "f4b41d60adfc4c0a8bd18700e5e96b52"
format_stamp: "Formatted at 2026-08-12 07:06:52 on dist-test-slave-69rg"
server_key: "5bbd94bffc42c29ea01f497815d7aebb"
server_key_iv: "70f709cb49b3771be7dc0d9bc3b593d6"
server_key_version: "kuduclusterkey@0"
I20260812 07:06:52.544791 20895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:52.565444 20895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:52.570401 20895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:52.570755 20895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:52.591714 20895 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:39677
I20260812 07:06:52.591744 20973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:39677 every 8 connection(s)
I20260812 07:06:52.593318 20895 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:06:52.594521 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 20895
I20260812 07:06:52.594779 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:06:52.595702 6388 external_mini_cluster.cc:1428] Setting key 7197be95d668e8b48a3563523ffd8491
I20260812 07:06:52.599519 20974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 07:06:52 dist-test-slave-69rg krb5kdc[20379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:06:52.613299 20974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52: Bootstrap starting.
I20260812 07:06:52.617076 20974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:52.618299 20974 log.cc:824] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:52.621840 20974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52: No bootstrap required, opened a new log
I20260812 07:06:52.623615 20977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:52.598105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57816 (local address 127.6.61.62:39677)
0812 07:06:52.598953 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:52.598968 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:52.599016 (+ 48us) server_negotiation.cc:407] Connection header received
0812 07:06:52.601292 (+ 2276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:52.601325 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:52.601809 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:52.602338 (+ 529us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:52.604027 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:52.606164 (+ 2137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:52.608087 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:52.609194 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:52.609385 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:52.614443 (+ 5058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:52.614454 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:52.614472 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:52.614517 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:52.618353 (+ 3836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:52.619317 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:52.619324 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:52.619331 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:52.619638 (+ 307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:52.620176 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:52.620181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:52.620184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:52.620905 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:52.621199 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:52.621706 (+ 507us) server_negotiation.cc:299] Negotiation successful
0812 07:06:52.622157 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":372,"threads_started":1}
I20260812 07:06:52.631249 20974 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b41d60adfc4c0a8bd18700e5e96b52" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39677 } }
I20260812 07:06:52.631628 20974 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:52.631762 20974 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4b41d60adfc4c0a8bd18700e5e96b52, State: Initialized, Role: FOLLOWER
I20260812 07:06:52.632319 20974 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b41d60adfc4c0a8bd18700e5e96b52" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39677 } }
I20260812 07:06:52.632566 20974 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:06:52.632719 20974 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:06:52.632975 20974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:52.635391 20974 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b41d60adfc4c0a8bd18700e5e96b52" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39677 } }
I20260812 07:06:52.635932 20974 leader_election.cc:302] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4b41d60adfc4c0a8bd18700e5e96b52; no voters:
I20260812 07:06:52.636631 20974 leader_election.cc:288] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:06:52.636777 20980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:06:52.641144 20980 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52 [term 1 LEADER]: Becoming Leader. State: Replica: f4b41d60adfc4c0a8bd18700e5e96b52, State: Running, Role: LEADER
I20260812 07:06:52.641871 20980 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b41d60adfc4c0a8bd18700e5e96b52" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39677 } }
I20260812 07:06:52.642351 20974 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:06:52.645177 20982 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4b41d60adfc4c0a8bd18700e5e96b52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b41d60adfc4c0a8bd18700e5e96b52" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39677 } } }
I20260812 07:06:52.645695 20982 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:52.646289 20983 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4b41d60adfc4c0a8bd18700e5e96b52. Latest consensus state: current_term: 1 leader_uuid: "f4b41d60adfc4c0a8bd18700e5e96b52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b41d60adfc4c0a8bd18700e5e96b52" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39677 } } }
I20260812 07:06:52.646639 20983 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:52.649721 20988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:06:52.657034 20988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:06:52.667961 20974 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 07:06:52.669417 20988 catalog_manager.cc:1347] Generated new cluster ID: 834622190f3944b985d497314bf8c700
I20260812 07:06:52.669523 20988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:06:52.705487 20988 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:06:52.707506 20988 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:06:52.717200 20988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52: Generated new TSK 0
I20260812 07:06:52.718366 20988 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
I20260812 07:06:53.546639 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.61.20:53803/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.1:0
--local_ip_for_outbound_sockets=127.6.61.1
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:39677
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 07:06:53 dist-test-slave-69rg krb5kdc[20379](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786518413, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260812 07:06:53.999626 21030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:54.000150 21030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:54.000342 21030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:54.009254 21030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:54.009428 21030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:54.009626 21030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.1
I20260812 07:06:54.020311 21030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.61.20:53803/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:39677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.1
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:06:54.022053 21030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:54.023761 21030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:54.044710 21041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:54.045449 21044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:54.046208 21042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:54.048606 21030 server_base.cc:1034] running on GCE node
I20260812 07:06:54.049451 21030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:54.061774 21030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:54.064004 21030 hybrid_clock.cc:648] HybridClock initialized: now 1786518414063773 us; error 134 us; skew 500 ppm
I20260812 07:06:54.065315 21030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:06:54 dist-test-slave-69rg krb5kdc[20379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518414, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:54.071653 21030 init.cc:377] Logged in from keytab as kudu/127.6.61.1@KRBTEST.COM (short username kudu)
I20260812 07:06:54.083290 21030 webserver.cc:460] Webserver started at http://127.6.61.1:35659/ using document root <none> and password file <none>
I20260812 07:06:54.084206 21030 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:54.084389 21030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:54.084733 21030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 07:06:54 dist-test-slave-69rg krb5kdc[20379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518414, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for HTTP/127.6.61.20@KRBTEST.COM
I20260812 07:06:54.105757 21030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-0/data/instance:
uuid: "f62d879e7e394f1385bf1a307796c52d"
format_stamp: "Formatted at 2026-08-12 07:06:54 on dist-test-slave-69rg"
server_key: "d05f567156805f8ae7860fa4cafbec21"
server_key_iv: "814c4fcd6b84ca001e7de26946223be6"
server_key_version: "kuduclusterkey@0"
I20260812 07:06:54.106956 21030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance:
uuid: "f62d879e7e394f1385bf1a307796c52d"
format_stamp: "Formatted at 2026-08-12 07:06:54 on dist-test-slave-69rg"
server_key: "d05f567156805f8ae7860fa4cafbec21"
server_key_iv: "814c4fcd6b84ca001e7de26946223be6"
server_key_version: "kuduclusterkey@0"
I20260812 07:06:54.117573 21030 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.007s sys 0.001s
I20260812 07:06:54.154587 21053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:54.156800 21030 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.011s sys 0.003s
I20260812 07:06:54.157087 21030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-0/wal
uuid: "f62d879e7e394f1385bf1a307796c52d"
format_stamp: "Formatted at 2026-08-12 07:06:54 on dist-test-slave-69rg"
server_key: "d05f567156805f8ae7860fa4cafbec21"
server_key_iv: "814c4fcd6b84ca001e7de26946223be6"
server_key_version: "kuduclusterkey@0"
I20260812 07:06:54.157476 21030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:54.192016 21030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:54.196578 21030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:54.197031 21030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:54.204988 21030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:06:54.207372 21030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:06:54.207480 21030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:54.207676 21030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:06:54.207765 21030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 07:06:54 dist-test-slave-69rg krb5kdc[20379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518413, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for HTTP/127.6.61.20@KRBTEST.COM
I20260812 07:06:54.325879 21030 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.1:42621
I20260812 07:06:54.325898 21191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.1:42621 every 8 connection(s)
I20260812 07:06:54.327466 21030 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
I20260812 07:06:54.332572 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 21030
I20260812 07:06:54.332854 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance
I20260812 07:06:54.333880 6388 external_mini_cluster.cc:1428] Setting key fa757c5b7caa75a0cdac258ee0d1c60b
Aug 12 07:06:54 dist-test-slave-69rg krb5kdc[20379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518414, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:06:54.369917 21195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:54.331560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39677 (local address 127.6.61.1:35997)
0812 07:06:54.346837 (+ 15277us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:54.346903 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:54.349524 (+ 2621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:54.350655 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:54.350667 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:54.351353 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:54.352244 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:54.352271 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:54.353888 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:54.353894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:54.355809 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:54.355825 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:54.357081 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:54.357087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:54.357371 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:54.358200 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:06:54.358235 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:54.360947 (+ 2712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:54.364820 (+ 3873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:54.364829 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:54.364849 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:54.365404 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:54.365938 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:54.365943 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:54.365946 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:54.366285 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:54.367004 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:54.367013 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:54.367484 (+ 471us) client_negotiation.cc:769] Sending connection context
0812 07:06:54.367829 (+ 345us) client_negotiation.cc:240] Negotiation successful
0812 07:06:54.368414 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14954,"thread_start_us":14693,"threads_started":1}
I20260812 07:06:54.371425 21194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:54.330786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:35997 (local address 127.6.61.62:39677)
0812 07:06:54.346187 (+ 15401us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:54.346198 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:54.347426 (+ 1228us) server_negotiation.cc:407] Connection header received
0812 07:06:54.349988 (+ 2562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:54.349995 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:54.350083 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:54.350204 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:54.352468 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:54.353676 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:54.355995 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:54.356656 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:54.356775 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:54.361243 (+ 4468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:54.361250 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:54.361255 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:54.361294 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:54.364651 (+ 3357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:54.365567 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:54.365573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:54.365575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:54.365749 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:54.366428 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:54.366432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:54.366435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:54.366785 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:54.366916 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:54.370808 (+ 3892us) server_negotiation.cc:299] Negotiation successful
0812 07:06:54.371048 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15182,"thread_start_us":16641,"threads_started":1}
I20260812 07:06:54.373780 21192 heartbeater.cc:347] Connected to a master server at 127.6.61.62:39677
I20260812 07:06:54.374151 21192 heartbeater.cc:464] Registering TS with master...
I20260812 07:06:54.375026 21192 heartbeater.cc:511] Master 127.6.61.62:39677 requested a full tablet report, sending...
I20260812 07:06:54.377660 20932 ts_manager.cc:194] Registered new tserver with Master: f62d879e7e394f1385bf1a307796c52d (127.6.61.1:42621)
I20260812 07:06:54.379797 20932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.1@KRBTEST.COM'} at 127.6.61.1:35997
WARNING: no policy specified for kudu/127.6.61.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
I20260812 07:06:54.429420 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.61.20:53803/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.2:0
--local_ip_for_outbound_sockets=127.6.61.2
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:39677
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 07:06:54 dist-test-slave-69rg krb5kdc[20379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518413, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for HTTP/127.6.61.20@KRBTEST.COM
W20260812 07:06:54.891593 21200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:54.891937 21200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:54.892045 21200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:54.900326 21200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:54.900450 21200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:54.900576 21200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.2
I20260812 07:06:54.911543 21200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.61.20:53803/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:39677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.2
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:06:54.912922 21200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:54.914398 21200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:54.928040 21218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:54.928295 21219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:54.928313 21222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:54.929512 21200 server_base.cc:1034] running on GCE node
I20260812 07:06:54.930258 21200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:54.931175 21200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:54.932395 21200 hybrid_clock.cc:648] HybridClock initialized: now 1786518414932352 us; error 32 us; skew 500 ppm
I20260812 07:06:54.932746 21200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:06:54 dist-test-slave-69rg krb5kdc[20379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518414, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:54.936146 21200 init.cc:377] Logged in from keytab as kudu/127.6.61.2@KRBTEST.COM (short username kudu)
I20260812 07:06:54.937891 21200 webserver.cc:460] Webserver started at http://127.6.61.2:46427/ using document root <none> and password file <none>
I20260812 07:06:54.938604 21200 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:54.938769 21200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:54.939101 21200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 07:06:54 dist-test-slave-69rg krb5kdc[20379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518414, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for HTTP/127.6.61.20@KRBTEST.COM
I20260812 07:06:54.954053 21200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-1/data/instance:
uuid: "80498ed6612544e0a5850c817065459d"
format_stamp: "Formatted at 2026-08-12 07:06:54 on dist-test-slave-69rg"
server_key: "a0141adc50828e7824a8d8d01e80029e"
server_key_iv: "a53fd0e43e586d3a80f3121d3eedbf2d"
server_key_version: "kuduclusterkey@0"
I20260812 07:06:54.954939 21200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance:
uuid: "80498ed6612544e0a5850c817065459d"
format_stamp: "Formatted at 2026-08-12 07:06:54 on dist-test-slave-69rg"
server_key: "a0141adc50828e7824a8d8d01e80029e"
server_key_iv: "a53fd0e43e586d3a80f3121d3eedbf2d"
server_key_version: "kuduclusterkey@0"
I20260812 07:06:54.960559 21200 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 07:06:54.978320 21229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:54.980067 21200 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 07:06:54.980301 21200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-1/wal
uuid: "80498ed6612544e0a5850c817065459d"
format_stamp: "Formatted at 2026-08-12 07:06:54 on dist-test-slave-69rg"
server_key: "a0141adc50828e7824a8d8d01e80029e"
server_key_iv: "a53fd0e43e586d3a80f3121d3eedbf2d"
server_key_version: "kuduclusterkey@0"
I20260812 07:06:54.980572 21200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:55.008417 21200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:55.013131 21200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:55.013489 21200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:55.014786 21200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:06:55.016428 21200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:06:55.016582 21200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:55.016743 21200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:06:55.016834 21200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:55.065322 21200 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.2:33413
I20260812 07:06:55.065327 21362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.2:33413 every 8 connection(s)
I20260812 07:06:55.066836 21200 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
I20260812 07:06:55.069198 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 21200
I20260812 07:06:55.069416 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance
I20260812 07:06:55.070129 6388 external_mini_cluster.cc:1428] Setting key 8a3e30f67aa8a4520e82f2fa34aa28b4
Aug 12 07:06:55 dist-test-slave-69rg krb5kdc[20379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518414, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:06:55.102032 21365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:55.070099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:53747 (local address 127.6.61.62:39677)
0812 07:06:55.082309 (+ 12210us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:55.082318 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:55.083538 (+ 1220us) server_negotiation.cc:407] Connection header received
0812 07:06:55.085850 (+ 2312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:55.085861 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:55.085988 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:55.086119 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:55.088221 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:55.089442 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:55.091315 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:55.091752 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:55.091908 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:55.095628 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:55.095634 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:55.095639 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:55.095693 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:55.098584 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:55.099357 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:55.099369 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:55.099373 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:55.099496 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:55.099981 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:55.099986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:55.099988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:55.100287 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:55.100465 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:55.101262 (+ 797us) server_negotiation.cc:299] Negotiation successful
0812 07:06:55.101597 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12027,"thread_start_us":11877,"threads_started":1}
I20260812 07:06:55.103112 21366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:55.070731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39677 (local address 127.6.61.2:53747)
0812 07:06:55.082799 (+ 12068us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:55.082862 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:55.085452 (+ 2590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:55.086444 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:55.086457 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:55.087121 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:55.087995 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:55.088023 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:55.089653 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:55.089658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:55.091132 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:55.091146 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:55.092009 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:55.092019 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:55.092310 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:55.093072 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:06:55.093096 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:55.095382 (+ 2286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:55.098727 (+ 3345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:55.098737 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:55.098758 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:55.099212 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:55.099615 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:55.099620 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:55.099624 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:55.099847 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:55.100426 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:55.100435 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:55.100826 (+ 391us) client_negotiation.cc:769] Sending connection context
0812 07:06:55.101164 (+ 338us) client_negotiation.cc:240] Negotiation successful
0812 07:06:55.101735 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11710,"thread_start_us":11438,"threads_started":1}
I20260812 07:06:55.104859 21363 heartbeater.cc:347] Connected to a master server at 127.6.61.62:39677
I20260812 07:06:55.105208 21363 heartbeater.cc:464] Registering TS with master...
I20260812 07:06:55.106072 21363 heartbeater.cc:511] Master 127.6.61.62:39677 requested a full tablet report, sending...
I20260812 07:06:55.107725 20932 ts_manager.cc:194] Registered new tserver with Master: 80498ed6612544e0a5850c817065459d (127.6.61.2:33413)
WARNING: no policy specified for kudu/127.6.61.3@KRBTEST.COM; defaulting to no policy
I20260812 07:06:55.108680 20932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.2@KRBTEST.COM'} at 127.6.61.2:53747
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
I20260812 07:06:55.145774 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.61.20:53803/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.3:0
--local_ip_for_outbound_sockets=127.6.61.3
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:39677
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:06:55.383832 21192 heartbeater.cc:503] Master 127.6.61.62:39677 was elected leader, sending a full tablet report...
W20260812 07:06:55.490618 21369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:55.490890 21369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:55.491055 21369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:55.498342 21369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:55.498452 21369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:55.498653 21369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.3
I20260812 07:06:55.508404 21369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.61.20:53803/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:39677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.3
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:06:55.509632 21369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:55.510985 21369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:55.524001 21376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:55.524057 21379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:55.523993 21377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:55.525307 21369 server_base.cc:1034] running on GCE node
I20260812 07:06:55.525966 21369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:55.526964 21369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:55.528198 21369 hybrid_clock.cc:648] HybridClock initialized: now 1786518415528156 us; error 24 us; skew 500 ppm
I20260812 07:06:55.528542 21369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:06:55 dist-test-slave-69rg krb5kdc[20379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518415, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:55.534909 21369 init.cc:377] Logged in from keytab as kudu/127.6.61.3@KRBTEST.COM (short username kudu)
I20260812 07:06:55.536296 21369 webserver.cc:460] Webserver started at http://127.6.61.3:39813/ using document root <none> and password file <none>
I20260812 07:06:55.536935 21369 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:55.537037 21369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:55.537294 21369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 07:06:55 dist-test-slave-69rg krb5kdc[20379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518415, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for HTTP/127.6.61.20@KRBTEST.COM
I20260812 07:06:55.553272 21369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-2/data/instance:
uuid: "767e1e4a42774ddb933b6fa833569080"
format_stamp: "Formatted at 2026-08-12 07:06:55 on dist-test-slave-69rg"
server_key: "474e0cac034dda0d7c0457a4dc512157"
server_key_iv: "3f3fddfe040df04aaf53d3259c68db41"
server_key_version: "kuduclusterkey@0"
I20260812 07:06:55.554414 21369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance:
uuid: "767e1e4a42774ddb933b6fa833569080"
format_stamp: "Formatted at 2026-08-12 07:06:55 on dist-test-slave-69rg"
server_key: "474e0cac034dda0d7c0457a4dc512157"
server_key_iv: "3f3fddfe040df04aaf53d3259c68db41"
server_key_version: "kuduclusterkey@0"
I20260812 07:06:55.561187 21369 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260812 07:06:55.577375 21387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:55.578920 21369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:06:55.579105 21369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-2/wal
uuid: "767e1e4a42774ddb933b6fa833569080"
format_stamp: "Formatted at 2026-08-12 07:06:55 on dist-test-slave-69rg"
server_key: "474e0cac034dda0d7c0457a4dc512157"
server_key_iv: "3f3fddfe040df04aaf53d3259c68db41"
server_key_version: "kuduclusterkey@0"
I20260812 07:06:55.579337 21369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:55.598417 21369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:55.603222 21369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:55.603487 21369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:55.604383 21369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:06:55.605743 21369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:06:55.605876 21369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260812 07:06:55.606034 21369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:06:55.606112 21369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:55.641661 21369 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.3:41569
I20260812 07:06:55.641709 21520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.3:41569 every 8 connection(s)
I20260812 07:06:55.643191 21369 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
I20260812 07:06:55.645565 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 21369
I20260812 07:06:55.645797 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance
I20260812 07:06:55.646520 6388 external_mini_cluster.cc:1428] Setting key 6d6426862967f027562e7d8ef67b0b7d
Aug 12 07:06:55 dist-test-slave-69rg krb5kdc[20379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518415, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:06:55.663954 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:55.645942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:51039 (local address 127.6.61.62:39677)
0812 07:06:55.646498 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:55.646504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:55.647849 (+ 1345us) server_negotiation.cc:407] Connection header received
0812 07:06:55.650111 (+ 2262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:55.650115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:55.650190 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:55.650279 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:55.651840 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:55.652562 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:55.654163 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:55.654578 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:55.654668 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:55.658026 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:55.658031 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:55.658034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:55.658065 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:55.660646 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:55.661530 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:55.661535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:55.661538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:55.661709 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:55.662255 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:55.662259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:55.662261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:55.662675 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:55.662800 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:55.663428 (+ 628us) server_negotiation.cc:299] Negotiation successful
0812 07:06:55.663686 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":256,"threads_started":1}
I20260812 07:06:55.664870 21526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:55.646430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39677 (local address 127.6.61.3:51039)
0812 07:06:55.647667 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:55.647724 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:55.649845 (+ 2121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:55.650516 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:55.650528 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:55.651081 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:55.651662 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:55.651680 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:55.652708 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:55.652713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:55.654035 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:55.654047 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:55.654770 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:55.654775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:55.654971 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:55.655668 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:06:55.655689 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:55.657759 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:55.660788 (+ 3029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:55.660796 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:55.660812 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:55.661375 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:55.661852 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:55.661857 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:55.661860 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:55.662124 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:55.662797 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:55.662803 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:55.663090 (+ 287us) client_negotiation.cc:769] Sending connection context
0812 07:06:55.663316 (+ 226us) client_negotiation.cc:240] Negotiation successful
0812 07:06:55.663800 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":955,"thread_start_us":729,"threads_started":1}
I20260812 07:06:55.666244 21521 heartbeater.cc:347] Connected to a master server at 127.6.61.62:39677
I20260812 07:06:55.666548 21521 heartbeater.cc:464] Registering TS with master...
I20260812 07:06:55.667260 21521 heartbeater.cc:511] Master 127.6.61.62:39677 requested a full tablet report, sending...
I20260812 07:06:55.668685 20932 ts_manager.cc:194] Registered new tserver with Master: 767e1e4a42774ddb933b6fa833569080 (127.6.61.3:41569)
I20260812 07:06:55.669440 20932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.3@KRBTEST.COM'} at 127.6.61.3:51039
I20260812 07:06:55.674758 6388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 07:06:55.676478 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 21030
Aug 12 07:06:55 dist-test-slave-69rg krb5kdc[20379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.1@KRBTEST.COM
I20260812 07:06:55.694473 21528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:55.677497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41964 (local address 127.6.61.1:42621)
0812 07:06:55.678243 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:55.678251 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:55.678288 (+ 37us) server_negotiation.cc:407] Connection header received
0812 07:06:55.678343 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:55.678349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:55.678535 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:55.678696 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:55.680306 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:55.681304 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:55.682942 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:55.683594 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:55.683744 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:55.688438 (+ 4694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:55.688450 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:55.688466 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:55.688501 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:55.691443 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:55.692155 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:55.692163 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:55.692165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:55.692380 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:55.692784 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:55.692794 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:55.692796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:55.693139 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:55.693479 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:55.693805 (+ 326us) server_negotiation.cc:299] Negotiation successful
0812 07:06:55.694114 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":349,"threads_started":1}
I20260812 07:06:55.695812 21186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:06:55.779726 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 21030
I20260812 07:06:55.801451 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 21200
Aug 12 07:06:55 dist-test-slave-69rg krb5kdc[20379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.2@KRBTEST.COM
I20260812 07:06:55.818403 21531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:55.802565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54396 (local address 127.6.61.2:33413)
0812 07:06:55.803162 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:55.803169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:55.803195 (+ 26us) server_negotiation.cc:407] Connection header received
0812 07:06:55.803272 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:55.803278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:55.803492 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:55.803623 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:55.804865 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:55.805900 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:55.807325 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:55.807844 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:55.807935 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:55.812524 (+ 4589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:55.812535 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:55.812548 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:55.812583 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:55.815482 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:55.816208 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:55.816216 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:55.816218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:55.816359 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:55.816753 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:55.816759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:55.816763 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:55.817105 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:55.817438 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:55.817731 (+ 293us) server_negotiation.cc:299] Negotiation successful
0812 07:06:55.818026 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":322,"threads_started":1}
I20260812 07:06:55.819695 21358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:06:55.902362 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 21200
I20260812 07:06:55.926928 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 21369
Aug 12 07:06:55 dist-test-slave-69rg krb5kdc[20379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.3@KRBTEST.COM
I20260812 07:06:55.944590 21533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:55.928097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37462 (local address 127.6.61.3:41569)
0812 07:06:55.928903 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:55.928910 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:55.928940 (+ 30us) server_negotiation.cc:407] Connection header received
0812 07:06:55.929001 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:55.929008 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:55.929237 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:55.929368 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:55.931133 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:55.932171 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:55.933750 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:55.934369 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:55.934470 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:55.938836 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:55.938848 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:55.938862 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:55.938898 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:55.941804 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:55.942456 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:55.942463 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:55.942465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:55.942590 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:55.943006 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:55.943009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:55.943011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:55.943346 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:55.943600 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:55.943938 (+ 338us) server_negotiation.cc:299] Negotiation successful
0812 07:06:55.944254 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":436,"threads_started":1}
I20260812 07:06:55.945966 21516 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:06:56.028358 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 21369
I20260812 07:06:56.049348 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 20895
I20260812 07:06:56.050899 20968 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:06:56.159490 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 20895
I20260812 07:06:56.187434 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:39677
--webserver_interface=127.6.61.62
--webserver_port=37369
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:39677
--ranger_config_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.61.20:53803/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.61.20:53803/kms with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:06:56.529035 21536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:56.529337 21536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:56.529428 21536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:56.536685 21536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:06:56.536798 21536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:56.536883 21536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:56.536988 21536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:06:56.537077 21536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:06:56.547370 21536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.61.20:53803/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:39677
--ranger_config_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:39677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=37369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:06:56.548663 21536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:56.550062 21536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:56.562026 21546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:56.561976 21544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:56.562067 21543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:56.562780 21536 server_base.cc:1034] running on GCE node
I20260812 07:06:56.563611 21536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:56.564980 21536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:56.566186 21536 hybrid_clock.cc:648] HybridClock initialized: now 1786518416566107 us; error 65 us; skew 500 ppm
I20260812 07:06:56.566540 21536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:06:56 dist-test-slave-69rg krb5kdc[20379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518416, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:56.569782 21536 init.cc:377] Logged in from keytab as kudu/127.6.61.62@KRBTEST.COM (short username kudu)
I20260812 07:06:56.571241 21536 webserver.cc:460] Webserver started at http://127.6.61.62:37369/ using document root <none> and password file <none>
I20260812 07:06:56.571903 21536 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:56.572047 21536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 07:06:56 dist-test-slave-69rg krb5kdc[20379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518416, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for HTTP/127.6.61.20@KRBTEST.COM
I20260812 07:06:56.627111 21536 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 07:06:56.630427 21556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:56.631906 21536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 07:06:56.632119 21536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "f4b41d60adfc4c0a8bd18700e5e96b52"
format_stamp: "Formatted at 2026-08-12 07:06:52 on dist-test-slave-69rg"
server_key: "5bbd94bffc42c29ea01f497815d7aebb"
server_key_iv: "70f709cb49b3771be7dc0d9bc3b593d6"
server_key_version: "kuduclusterkey@0"
I20260812 07:06:56.633114 21536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:56.657332 21536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:56.661862 21536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:56.662235 21536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:56.683637 21536 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:39677
I20260812 07:06:56.683661 21621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:39677 every 8 connection(s)
I20260812 07:06:56.685293 21536 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:06:56.687382 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 21536
I20260812 07:06:56.688469 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.61.20:53803/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.1:42621
--local_ip_for_outbound_sockets=127.6.61.1
--tserver_master_addrs=127.6.61.62:39677
--webserver_port=35659
--webserver_interface=127.6.61.1
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.61.20:53803/kms with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:06:56.695226 21622 sys_catalog.cc:263] Verifying existing consensus state
I20260812 07:06:56.700444 21622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52: Bootstrap starting.
I20260812 07:06:56.720563 21622 log.cc:824] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:56.729707 21622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 07:06:56.730366 21622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52: Bootstrap complete.
I20260812 07:06:56.739974 21622 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b41d60adfc4c0a8bd18700e5e96b52" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39677 } }
I20260812 07:06:56.740868 21622 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4b41d60adfc4c0a8bd18700e5e96b52, State: Initialized, Role: FOLLOWER
I20260812 07:06:56.741552 21622 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b41d60adfc4c0a8bd18700e5e96b52" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39677 } }
I20260812 07:06:56.741811 21622 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:06:56.741989 21622 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:06:56.742236 21622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52 [term 1 FOLLOWER]: Advancing to term 2
I20260812 07:06:56.746354 21622 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b41d60adfc4c0a8bd18700e5e96b52" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39677 } }
I20260812 07:06:56.746855 21622 leader_election.cc:302] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4b41d60adfc4c0a8bd18700e5e96b52; no voters:
I20260812 07:06:56.747555 21622 leader_election.cc:288] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 07:06:56.747706 21626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 07:06:56.748935 21626 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52 [term 2 LEADER]: Becoming Leader. State: Replica: f4b41d60adfc4c0a8bd18700e5e96b52, State: Running, Role: LEADER
I20260812 07:06:56.749547 21626 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b41d60adfc4c0a8bd18700e5e96b52" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39677 } }
I20260812 07:06:56.749912 21622 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:06:56.752278 21627 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f4b41d60adfc4c0a8bd18700e5e96b52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b41d60adfc4c0a8bd18700e5e96b52" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39677 } } }
I20260812 07:06:56.752740 21627 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:56.754151 21628 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4b41d60adfc4c0a8bd18700e5e96b52. Latest consensus state: current_term: 2 leader_uuid: "f4b41d60adfc4c0a8bd18700e5e96b52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b41d60adfc4c0a8bd18700e5e96b52" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39677 } } }
I20260812 07:06:56.754547 21628 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:56.755631 21631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:06:56.761674 21631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:06:56.763149 21631 catalog_manager.cc:1259] Loaded cluster ID: 834622190f3944b985d497314bf8c700
I20260812 07:06:56.763317 21631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:06:56.767374 21631 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:06:56.768836 21631 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52: Loaded TSK: 0
I20260812 07:06:56.770463 21631 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:06:56.780182 21622 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260812 07:06:57.137094 21624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:57.137447 21624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:57.137653 21624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:57.149210 21624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:57.149438 21624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:57.149677 21624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.1
I20260812 07:06:57.165292 21624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.61.20:53803/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.1:42621
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.61.1
--webserver_port=35659
--tserver_master_addrs=127.6.61.62:39677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.1
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:06:57.167129 21624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:57.168885 21624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:57.189051 21662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:57.194232 21663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:57.194844 21624 server_base.cc:1034] running on GCE node
W20260812 07:06:57.195349 21666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:57.196257 21624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:57.198642 21624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:57.202073 21624 hybrid_clock.cc:648] HybridClock initialized: now 1786518417201770 us; error 281 us; skew 500 ppm
I20260812 07:06:57.202481 21624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:06:57 dist-test-slave-69rg krb5kdc[20379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518417, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:57.207231 21624 init.cc:377] Logged in from keytab as kudu/127.6.61.1@KRBTEST.COM (short username kudu)
I20260812 07:06:57.215390 21624 webserver.cc:460] Webserver started at http://127.6.61.1:35659/ using document root <none> and password file <none>
I20260812 07:06:57.216297 21624 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:57.216480 21624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 07:06:57 dist-test-slave-69rg krb5kdc[20379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518417, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for HTTP/127.6.61.20@KRBTEST.COM
I20260812 07:06:57.254674 21624 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260812 07:06:57.267690 21675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:57.270465 21624 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.006s sys 0.005s
I20260812 07:06:57.270920 21624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-0/wal
uuid: "f62d879e7e394f1385bf1a307796c52d"
format_stamp: "Formatted at 2026-08-12 07:06:54 on dist-test-slave-69rg"
server_key: "d05f567156805f8ae7860fa4cafbec21"
server_key_iv: "814c4fcd6b84ca001e7de26946223be6"
server_key_version: "kuduclusterkey@0"
I20260812 07:06:57.272524 21624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:57.308915 21624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:57.314461 21624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:57.314805 21624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:57.316146 21624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:06:57.322721 21624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:06:57.322858 21624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:57.323022 21624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:06:57.323101 21624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:57.431345 21624 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.1:42621
I20260812 07:06:57.431324 21804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.1:42621 every 8 connection(s)
I20260812 07:06:57.433182 21624 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
I20260812 07:06:57.434857 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 21624
I20260812 07:06:57.435703 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.61.20:53803/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.2:33413
--local_ip_for_outbound_sockets=127.6.61.2
--tserver_master_addrs=127.6.61.62:39677
--webserver_port=46427
--webserver_interface=127.6.61.2
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.61.20:53803/kms with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 07:06:57 dist-test-slave-69rg krb5kdc[20379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518417, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:06:57.481886 21808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.436891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:49025 (local address 127.6.61.62:39677)
0812 07:06:57.450691 (+ 13800us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:57.450710 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:57.450763 (+ 53us) server_negotiation.cc:407] Connection header received
0812 07:06:57.451215 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:57.451235 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:57.451769 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:57.452179 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:57.455329 (+ 3150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.457402 (+ 2073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.459782 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.460750 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.460965 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:57.470259 (+ 9294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:57.470273 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:57.470299 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:57.470359 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:57.475080 (+ 4721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:57.476259 (+ 1179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:57.476272 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:57.476284 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:57.476509 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:57.477162 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:57.477170 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:57.477176 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:57.478022 (+ 846us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:57.478320 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:57.479621 (+ 1301us) server_negotiation.cc:299] Negotiation successful
0812 07:06:57.480160 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13530,"thread_start_us":13276,"threads_started":1}
I20260812 07:06:57.485150 21807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.436862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39677 (local address 127.6.61.1:49025)
0812 07:06:57.446063 (+ 9201us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:57.446142 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:57.448834 (+ 2692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:57.453486 (+ 4652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:57.453498 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:57.454220 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:57.455044 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:57.455073 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.457615 (+ 2542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.457623 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:57.459559 (+ 1936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:57.459579 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.461012 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.461020 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:57.461317 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:57.465001 (+ 3684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:06:57.465038 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:57.469797 (+ 4759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:57.475298 (+ 5501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:57.475310 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:57.475329 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:57.476041 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:57.476667 (+ 626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:57.476674 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:57.476677 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:57.476995 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:57.478410 (+ 1415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:57.478423 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:57.478916 (+ 493us) client_negotiation.cc:769] Sending connection context
0812 07:06:57.482967 (+ 4051us) client_negotiation.cc:240] Negotiation successful
0812 07:06:57.483614 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8788,"thread_start_us":12323,"threads_started":1}
I20260812 07:06:57.489353 21805 heartbeater.cc:347] Connected to a master server at 127.6.61.62:39677
I20260812 07:06:57.489691 21805 heartbeater.cc:464] Registering TS with master...
I20260812 07:06:57.490613 21805 heartbeater.cc:511] Master 127.6.61.62:39677 requested a full tablet report, sending...
W20260812 07:06:57.494109 21805 heartbeater.cc:644] Failed to heartbeat to 127.6.61.62:39677 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 07:06:57.494381 21805 heartbeater.cc:464] Registering TS with master...
I20260812 07:06:57.494689 21805 heartbeater.cc:511] Master 127.6.61.62:39677 requested a full tablet report, sending...
I20260812 07:06:57.495996 21805 heartbeater.cc:464] Registering TS with master...
I20260812 07:06:57.496321 21805 heartbeater.cc:511] Master 127.6.61.62:39677 requested a full tablet report, sending...
W20260812 07:06:57.497491 21805 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 07:06:57.855057 21809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:57.855448 21809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:57.855611 21809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:57.863399 21809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:57.863548 21809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:57.863718 21809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.2
I20260812 07:06:57.874594 21809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.61.20:53803/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.2:33413
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.61.2
--webserver_port=46427
--tserver_master_addrs=127.6.61.62:39677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.2
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:06:57.875990 21809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:57.877346 21809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:57.893522 21823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:57.894377 21824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:57.894922 21809 server_base.cc:1034] running on GCE node
W20260812 07:06:57.896319 21826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:57.897754 21809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:57.899726 21809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:57.901003 21809 hybrid_clock.cc:648] HybridClock initialized: now 1786518417900939 us; error 50 us; skew 500 ppm
I20260812 07:06:57.901427 21809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:06:57 dist-test-slave-69rg krb5kdc[20379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518417, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:57.905620 21809 init.cc:377] Logged in from keytab as kudu/127.6.61.2@KRBTEST.COM (short username kudu)
I20260812 07:06:57.910195 21809 webserver.cc:460] Webserver started at http://127.6.61.2:46427/ using document root <none> and password file <none>
I20260812 07:06:57.911051 21809 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:57.911233 21809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 07:06:57 dist-test-slave-69rg krb5kdc[20379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518417, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for HTTP/127.6.61.20@KRBTEST.COM
I20260812 07:06:57.935235 21809 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 07:06:57.938673 21839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:57.940373 21809 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 07:06:57.940575 21809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-1/wal
uuid: "80498ed6612544e0a5850c817065459d"
format_stamp: "Formatted at 2026-08-12 07:06:54 on dist-test-slave-69rg"
server_key: "a0141adc50828e7824a8d8d01e80029e"
server_key_iv: "a53fd0e43e586d3a80f3121d3eedbf2d"
server_key_version: "kuduclusterkey@0"
I20260812 07:06:57.941648 21809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:58.013378 21809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:58.019230 21809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:58.019651 21809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:58.021888 21809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:06:58.023854 21809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:06:58.023990 21809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:58.024119 21809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:06:58.024207 21809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:58.091516 21809 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.2:33413
I20260812 07:06:58.091578 21975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.2:33413 every 8 connection(s)
I20260812 07:06:58.093039 21809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
Aug 12 07:06:58 dist-test-slave-69rg krb5kdc[20379](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786518418, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:58.100026 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 21809
I20260812 07:06:58.100919 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.61.20:53803/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.3:41569
--local_ip_for_outbound_sockets=127.6.61.3
--tserver_master_addrs=127.6.61.62:39677
--webserver_port=39813
--webserver_interface=127.6.61.3
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.61.20:53803/kms with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 07:06:58 dist-test-slave-69rg krb5kdc[20379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518417, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:06:58.127476 21980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:58.098144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:36875 (local address 127.6.61.62:39677)
0812 07:06:58.103815 (+ 5671us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:58.103824 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:58.103852 (+ 28us) server_negotiation.cc:407] Connection header received
0812 07:06:58.103931 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:58.103939 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:58.104048 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:58.104167 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:58.108473 (+ 4306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:58.109726 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:58.112484 (+ 2758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:58.113239 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:58.113364 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:58.118709 (+ 5345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:58.118719 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:58.118727 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:58.118778 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:58.122977 (+ 4199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:58.124063 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:58.124072 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:58.124078 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:58.124265 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:58.124917 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:58.124925 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:58.124931 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:58.125356 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:58.125485 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:58.126717 (+ 1232us) server_negotiation.cc:299] Negotiation successful
0812 07:06:58.126987 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5478,"thread_start_us":7972,"threads_started":1}
I20260812 07:06:58.130386 21979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:58.097124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39677 (local address 127.6.61.2:36875)
0812 07:06:58.098752 (+ 1628us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:58.098816 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:58.101520 (+ 2704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:58.106497 (+ 4977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:58.106512 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:58.107268 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:58.108218 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:58.108249 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:58.110001 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:58.110008 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:58.112283 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:58.112304 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:58.113780 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:58.113790 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:58.114157 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:58.115025 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:06:58.115071 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:58.118389 (+ 3318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:58.123212 (+ 4823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:58.123225 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:58.123247 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:58.123912 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:58.124430 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:58.124438 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:58.124444 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:58.124769 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:58.125596 (+ 827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:58.125606 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:58.126129 (+ 523us) client_negotiation.cc:769] Sending connection context
0812 07:06:58.128083 (+ 1954us) client_negotiation.cc:240] Negotiation successful
0812 07:06:58.128772 (+ 689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1277,"thread_start_us":1065,"threads_started":1}
I20260812 07:06:58.132683 21976 heartbeater.cc:347] Connected to a master server at 127.6.61.62:39677
I20260812 07:06:58.133038 21976 heartbeater.cc:464] Registering TS with master...
I20260812 07:06:58.134151 21976 heartbeater.cc:511] Master 127.6.61.62:39677 requested a full tablet report, sending...
I20260812 07:06:58.136505 21578 ts_manager.cc:194] Registered new tserver with Master: 80498ed6612544e0a5850c817065459d (127.6.61.2:33413)
I20260812 07:06:58.138885 21578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.2@KRBTEST.COM'} at 127.6.61.2:36875
I20260812 07:06:58.497676 21805 heartbeater.cc:464] Registering TS with master...
I20260812 07:06:58.498291 21805 heartbeater.cc:511] Master 127.6.61.62:39677 requested a full tablet report, sending...
I20260812 07:06:58.499655 21578 ts_manager.cc:194] Registered new tserver with Master: f62d879e7e394f1385bf1a307796c52d (127.6.61.1:42621)
I20260812 07:06:58.500841 21578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.1@KRBTEST.COM'} at 127.6.61.1:49025
W20260812 07:06:58.523083 21981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:58.523437 21981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:58.523629 21981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:58.531596 21981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:58.531754 21981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:58.531951 21981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.3
I20260812 07:06:58.542847 21981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.61.20:53803/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.3:41569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.61.3
--webserver_port=39813
--tserver_master_addrs=127.6.61.62:39677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.3
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:06:58.544692 21981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:58.546558 21981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:58.565498 21990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:58.570164 21992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:58.570389 21981 server_base.cc:1034] running on GCE node
W20260812 07:06:58.570647 21988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:58.571497 21981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:58.572711 21981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:58.573968 21981 hybrid_clock.cc:648] HybridClock initialized: now 1786518418573896 us; error 71 us; skew 500 ppm
I20260812 07:06:58.574333 21981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:06:58 dist-test-slave-69rg krb5kdc[20379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518418, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:58.577756 21981 init.cc:377] Logged in from keytab as kudu/127.6.61.3@KRBTEST.COM (short username kudu)
I20260812 07:06:58.579875 21981 webserver.cc:460] Webserver started at http://127.6.61.3:39813/ using document root <none> and password file <none>
I20260812 07:06:58.580624 21981 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:58.580806 21981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 07:06:58 dist-test-slave-69rg krb5kdc[20379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518418, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for HTTP/127.6.61.20@KRBTEST.COM
Aug 12 07:06:58 dist-test-slave-69rg krb5kdc[20379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518418, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for HTTP/127.6.61.20@KRBTEST.COM
I20260812 07:06:58.648833 21981 fs_manager.cc:709] Time spent opening directory manager: real 0.038s user 0.023s sys 0.014s
I20260812 07:06:58.662496 22000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:58.664749 21981 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.005s sys 0.004s
I20260812 07:06:58.665055 21981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-2/wal
uuid: "767e1e4a42774ddb933b6fa833569080"
format_stamp: "Formatted at 2026-08-12 07:06:55 on dist-test-slave-69rg"
server_key: "474e0cac034dda0d7c0457a4dc512157"
server_key_iv: "3f3fddfe040df04aaf53d3259c68db41"
server_key_version: "kuduclusterkey@0"
I20260812 07:06:58.666694 21981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:58.695750 21981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:58.701540 21981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:58.701934 21981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:58.704599 21981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:06:58.706444 21981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:06:58.706575 21981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:58.706687 21981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:06:58.706810 21981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260812 07:06:58.756872 21981 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.3:41569
I20260812 07:06:58.756968 22135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.3:41569 every 8 connection(s)
I20260812 07:06:58.758812 21981 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
I20260812 07:06:58.767194 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 21981
Aug 12 07:06:58 dist-test-slave-69rg krb5kdc[20379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518418, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:06:58.807005 22140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:58.762385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39677 (local address 127.6.61.3:58373)
0812 07:06:58.776888 (+ 14503us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:58.776949 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:58.780704 (+ 3755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:58.783116 (+ 2412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:58.783129 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:58.783779 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:58.784597 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:58.784621 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:58.786656 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:58.786666 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:58.789416 (+ 2750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:58.789438 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:58.792544 (+ 3106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:58.792554 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:58.792956 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:58.794748 (+ 1792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:06:58.794780 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:58.797445 (+ 2665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:58.801784 (+ 4339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:58.801797 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:58.801852 (+ 55us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:58.802489 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:58.802966 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:58.802971 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:58.802975 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:58.803270 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:58.804032 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:58.804041 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:58.804521 (+ 480us) client_negotiation.cc:769] Sending connection context
0812 07:06:58.804885 (+ 364us) client_negotiation.cc:240] Negotiation successful
0812 07:06:58.805500 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14123,"thread_start_us":15823,"threads_started":1}
I20260812 07:06:58.808254 22139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:58.761742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:58373 (local address 127.6.61.62:39677)
0812 07:06:58.781898 (+ 20156us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:58.781908 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:58.781937 (+ 29us) server_negotiation.cc:407] Connection header received
0812 07:06:58.782027 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:58.782036 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:58.782178 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:58.782347 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:58.784771 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:58.786377 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:58.790431 (+ 4054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:58.791374 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:58.791533 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:58.797746 (+ 6213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:58.797755 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:58.797761 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:58.797856 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:58.801596 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:58.802647 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:58.802653 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:58.802657 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:58.802818 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:58.803425 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:58.803430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:58.803433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:58.803790 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:58.803943 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:58.807592 (+ 3649us) server_negotiation.cc:299] Negotiation successful
0812 07:06:58.807835 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19972,"thread_start_us":19854,"threads_started":1}
I20260812 07:06:58.810323 22136 heartbeater.cc:347] Connected to a master server at 127.6.61.62:39677
I20260812 07:06:58.810678 22136 heartbeater.cc:464] Registering TS with master...
I20260812 07:06:58.811573 22136 heartbeater.cc:511] Master 127.6.61.62:39677 requested a full tablet report, sending...
I20260812 07:06:58.813446 21578 ts_manager.cc:194] Registered new tserver with Master: 767e1e4a42774ddb933b6fa833569080 (127.6.61.3:41569)
I20260812 07:06:58.814687 21578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.3@KRBTEST.COM'} at 127.6.61.3:58373
I20260812 07:06:58.815385 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:58.768345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57828 (local address 127.6.61.62:39677)
0812 07:06:58.795028 (+ 26683us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:58.795034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:58.795051 (+ 17us) server_negotiation.cc:407] Connection header received
0812 07:06:58.795098 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:58.795102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:58.795173 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:58.795274 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:58.798133 (+ 2859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:58.799341 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:58.801297 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:58.801983 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:58.802079 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:58.805479 (+ 3400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:58.805486 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:58.805490 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:58.805526 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:58.809575 (+ 4049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:58.812520 (+ 2945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:58.812526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:58.812530 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:58.812682 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:58.813193 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:58.813201 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:58.813205 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:58.813640 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:58.814016 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:58.814715 (+ 699us) server_negotiation.cc:299] Negotiation successful
0812 07:06:58.815027 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26562,"thread_start_us":26384,"threads_started":1}
I20260812 07:06:58.819352 6388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 07:06:58.844165 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:58.822849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57834 (local address 127.6.61.62:39677)
0812 07:06:58.823057 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:58.823065 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:58.830469 (+ 7404us) server_negotiation.cc:407] Connection header received
0812 07:06:58.830695 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:58.830707 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:58.830777 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:58.830896 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:58.832468 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:58.833201 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:58.834831 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:58.835240 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:58.835388 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:58.838441 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:58.838446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:58.838449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:58.838485 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:58.841190 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:58.842062 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:58.842067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:58.842069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:58.842203 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:58.842619 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:58.842623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:58.842626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:58.842920 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:58.843180 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:58.843560 (+ 380us) server_negotiation.cc:299] Negotiation successful
0812 07:06:58.843824 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 07:06:58.849646 21578 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:06:58.858036 21578 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:06:58.928489 22157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:58.879469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:33413 (local address 127.0.0.1:54410)
0812 07:06:58.890094 (+ 10625us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:58.890155 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:58.890361 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:58.920131 (+ 29770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:58.920140 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:58.920180 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:58.920711 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:58.920730 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:58.922692 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:58.922697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:58.925186 (+ 2489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:58.925204 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:58.927487 (+ 2283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:58.927494 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:58.927722 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:58.927782 (+ 60us) client_negotiation.cc:769] Sending connection context
0812 07:06:58.927926 (+ 144us) client_negotiation.cc:240] Negotiation successful
0812 07:06:58.928123 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10279,"thread_start_us":11099,"threads_started":1}
I20260812 07:06:58.930176 22160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:58.897682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:41569 (local address 127.0.0.1:37476)
0812 07:06:58.905211 (+ 7529us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:58.905233 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:58.905365 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:58.923263 (+ 17898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:58.923267 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:58.923292 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:58.923691 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:58.923703 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:58.925587 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:58.925592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:58.927518 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:58.927535 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:58.929463 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:58.929468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:58.929615 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:58.929658 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 07:06:58.929740 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 07:06:58.929881 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7386,"thread_start_us":7758,"threads_started":1}
I20260812 07:06:58.930456 22158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:58.879856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54410 (local address 127.6.61.2:33413)
0812 07:06:58.919328 (+ 39472us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:58.919337 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:58.919374 (+ 37us) server_negotiation.cc:407] Connection header received
0812 07:06:58.919490 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:58.919498 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:58.919823 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:58.920071 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:58.920945 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:58.922504 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:58.925415 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:58.926946 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:58.927084 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:58.927302 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:58.929958 (+ 2656us) server_negotiation.cc:299] Negotiation successful
0812 07:06:58.930136 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39132,"thread_start_us":39540,"threads_started":1}
I20260812 07:06:58.932505 22159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:58.891722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37476 (local address 127.6.61.3:41569)
0812 07:06:58.922395 (+ 30673us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:58.922403 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:58.922436 (+ 33us) server_negotiation.cc:407] Connection header received
0812 07:06:58.922528 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:58.922535 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:58.922786 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:58.922977 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:58.923919 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:58.925419 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:58.927719 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:58.929006 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:58.929126 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:58.929322 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:58.932006 (+ 2684us) server_negotiation.cc:299] Negotiation successful
0812 07:06:58.932183 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30011,"thread_start_us":30213,"threads_started":1}
I20260812 07:06:58.937319 22157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:58.920708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:42621 (local address 127.0.0.1:41978)
0812 07:06:58.929018 (+ 8310us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:58.929039 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:58.929153 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:58.931079 (+ 1926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:58.931084 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:58.931114 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:58.931289 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:58.931300 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:58.933190 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:58.933196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:58.934791 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:58.934804 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:58.936553 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:58.936559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:58.936749 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:58.936787 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 07:06:58.936883 (+ 96us) client_negotiation.cc:240] Negotiation successful
0812 07:06:58.937030 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8208,"thread_start_us":10745,"threads_started":1}
I20260812 07:06:58.937477 22161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:58.906194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41978 (local address 127.6.61.1:42621)
0812 07:06:58.918564 (+ 12370us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:58.918574 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:58.929293 (+ 10719us) server_negotiation.cc:407] Connection header received
0812 07:06:58.929410 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:58.929418 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:58.929694 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:58.930807 (+ 1113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:58.931518 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:58.933029 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:58.934971 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:58.936387 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:58.936584 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:58.936825 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:58.936923 (+ 98us) server_negotiation.cc:299] Negotiation successful
0812 07:06:58.937079 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12161,"thread_start_us":12394,"threads_started":1}
I20260812 07:06:58.937088 22060 tablet_service.cc:1467] Processing CreateTablet for tablet 1290f70ea9594880b91405e5f04c37c1 (DEFAULT_TABLE table=test-table [id=2bf1a3b9ac1444cea5a2ef8de25a3bcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:06:58.938818 22060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1290f70ea9594880b91405e5f04c37c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:58.938562 21897 tablet_service.cc:1467] Processing CreateTablet for tablet 1290f70ea9594880b91405e5f04c37c1 (DEFAULT_TABLE table=test-table [id=2bf1a3b9ac1444cea5a2ef8de25a3bcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:06:58.939872 21897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1290f70ea9594880b91405e5f04c37c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:58.945760 21727 tablet_service.cc:1467] Processing CreateTablet for tablet 1290f70ea9594880b91405e5f04c37c1 (DEFAULT_TABLE table=test-table [id=2bf1a3b9ac1444cea5a2ef8de25a3bcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:06:58.947448 21727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1290f70ea9594880b91405e5f04c37c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:58.962329 22166 tablet_bootstrap.cc:492] T 1290f70ea9594880b91405e5f04c37c1 P 80498ed6612544e0a5850c817065459d: Bootstrap starting.
I20260812 07:06:58.968432 22166 tablet_bootstrap.cc:654] T 1290f70ea9594880b91405e5f04c37c1 P 80498ed6612544e0a5850c817065459d: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:58.969940 22166 log.cc:824] T 1290f70ea9594880b91405e5f04c37c1 P 80498ed6612544e0a5850c817065459d: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:58.981979 22169 tablet_bootstrap.cc:492] T 1290f70ea9594880b91405e5f04c37c1 P 767e1e4a42774ddb933b6fa833569080: Bootstrap starting.
I20260812 07:06:58.983908 22168 tablet_bootstrap.cc:492] T 1290f70ea9594880b91405e5f04c37c1 P f62d879e7e394f1385bf1a307796c52d: Bootstrap starting.
I20260812 07:06:58.988049 22169 tablet_bootstrap.cc:654] T 1290f70ea9594880b91405e5f04c37c1 P 767e1e4a42774ddb933b6fa833569080: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:58.988566 22168 tablet_bootstrap.cc:654] T 1290f70ea9594880b91405e5f04c37c1 P f62d879e7e394f1385bf1a307796c52d: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:58.990000 22168 log.cc:824] T 1290f70ea9594880b91405e5f04c37c1 P f62d879e7e394f1385bf1a307796c52d: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:58.991130 22169 log.cc:824] T 1290f70ea9594880b91405e5f04c37c1 P 767e1e4a42774ddb933b6fa833569080: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:58.993007 22166 tablet_bootstrap.cc:492] T 1290f70ea9594880b91405e5f04c37c1 P 80498ed6612544e0a5850c817065459d: No bootstrap required, opened a new log
I20260812 07:06:58.993399 22166 ts_tablet_manager.cc:1397] T 1290f70ea9594880b91405e5f04c37c1 P 80498ed6612544e0a5850c817065459d: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.000s
Aug 12 07:06:58 dist-test-slave-69rg krb5kdc[20379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518418, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for HTTP/127.6.61.20@KRBTEST.COM
I20260812 07:06:59.000134 22168 tablet_bootstrap.cc:492] T 1290f70ea9594880b91405e5f04c37c1 P f62d879e7e394f1385bf1a307796c52d: No bootstrap required, opened a new log
I20260812 07:06:59.000485 22168 ts_tablet_manager.cc:1397] T 1290f70ea9594880b91405e5f04c37c1 P f62d879e7e394f1385bf1a307796c52d: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.005s
I20260812 07:06:59.005111 22166 raft_consensus.cc:348] T 1290f70ea9594880b91405e5f04c37c1 P 80498ed6612544e0a5850c817065459d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80498ed6612544e0a5850c817065459d" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 33413 } } peers { permanent_uuid: "767e1e4a42774ddb933b6fa833569080" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 41569 } } peers { permanent_uuid: "f62d879e7e394f1385bf1a307796c52d" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42621 } }
I20260812 07:06:59.005653 22166 raft_consensus.cc:374] T 1290f70ea9594880b91405e5f04c37c1 P 80498ed6612544e0a5850c817065459d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:59.005805 22166 raft_consensus.cc:729] T 1290f70ea9594880b91405e5f04c37c1 P 80498ed6612544e0a5850c817065459d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80498ed6612544e0a5850c817065459d, State: Initialized, Role: FOLLOWER
I20260812 07:06:59.006632 22166 consensus_queue.cc:260] T 1290f70ea9594880b91405e5f04c37c1 P 80498ed6612544e0a5850c817065459d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80498ed6612544e0a5850c817065459d" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 33413 } } peers { permanent_uuid: "767e1e4a42774ddb933b6fa833569080" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 41569 } } peers { permanent_uuid: "f62d879e7e394f1385bf1a307796c52d" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42621 } }
I20260812 07:06:59.008770 22169 tablet_bootstrap.cc:492] T 1290f70ea9594880b91405e5f04c37c1 P 767e1e4a42774ddb933b6fa833569080: No bootstrap required, opened a new log
I20260812 07:06:59.009132 22169 ts_tablet_manager.cc:1397] T 1290f70ea9594880b91405e5f04c37c1 P 767e1e4a42774ddb933b6fa833569080: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.000s
I20260812 07:06:59.020710 22169 raft_consensus.cc:348] T 1290f70ea9594880b91405e5f04c37c1 P 767e1e4a42774ddb933b6fa833569080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80498ed6612544e0a5850c817065459d" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 33413 } } peers { permanent_uuid: "767e1e4a42774ddb933b6fa833569080" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 41569 } } peers { permanent_uuid: "f62d879e7e394f1385bf1a307796c52d" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42621 } }
I20260812 07:06:59.020781 22168 raft_consensus.cc:348] T 1290f70ea9594880b91405e5f04c37c1 P f62d879e7e394f1385bf1a307796c52d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80498ed6612544e0a5850c817065459d" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 33413 } } peers { permanent_uuid: "767e1e4a42774ddb933b6fa833569080" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 41569 } } peers { permanent_uuid: "f62d879e7e394f1385bf1a307796c52d" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42621 } }
I20260812 07:06:59.021301 22168 raft_consensus.cc:374] T 1290f70ea9594880b91405e5f04c37c1 P f62d879e7e394f1385bf1a307796c52d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:59.021306 22169 raft_consensus.cc:374] T 1290f70ea9594880b91405e5f04c37c1 P 767e1e4a42774ddb933b6fa833569080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:59.021435 22168 raft_consensus.cc:729] T 1290f70ea9594880b91405e5f04c37c1 P f62d879e7e394f1385bf1a307796c52d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f62d879e7e394f1385bf1a307796c52d, State: Initialized, Role: FOLLOWER
I20260812 07:06:59.021443 22169 raft_consensus.cc:729] T 1290f70ea9594880b91405e5f04c37c1 P 767e1e4a42774ddb933b6fa833569080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 767e1e4a42774ddb933b6fa833569080, State: Initialized, Role: FOLLOWER
I20260812 07:06:59.022217 22169 consensus_queue.cc:260] T 1290f70ea9594880b91405e5f04c37c1 P 767e1e4a42774ddb933b6fa833569080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80498ed6612544e0a5850c817065459d" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 33413 } } peers { permanent_uuid: "767e1e4a42774ddb933b6fa833569080" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 41569 } } peers { permanent_uuid: "f62d879e7e394f1385bf1a307796c52d" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42621 } }
I20260812 07:06:59.022219 22168 consensus_queue.cc:260] T 1290f70ea9594880b91405e5f04c37c1 P f62d879e7e394f1385bf1a307796c52d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80498ed6612544e0a5850c817065459d" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 33413 } } peers { permanent_uuid: "767e1e4a42774ddb933b6fa833569080" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 41569 } } peers { permanent_uuid: "f62d879e7e394f1385bf1a307796c52d" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42621 } }
I20260812 07:06:59.027531 22169 ts_tablet_manager.cc:1428] T 1290f70ea9594880b91405e5f04c37c1 P 767e1e4a42774ddb933b6fa833569080: Time spent starting tablet: real 0.018s user 0.009s sys 0.008s
I20260812 07:06:59.027953 22136 heartbeater.cc:503] Master 127.6.61.62:39677 was elected leader, sending a full tablet report...
W20260812 07:06:59.033349 22137 tablet.cc:2369] T 1290f70ea9594880b91405e5f04c37c1 P 767e1e4a42774ddb933b6fa833569080: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:06:59.036567 21805 heartbeater.cc:503] Master 127.6.61.62:39677 was elected leader, sending a full tablet report...
I20260812 07:06:59.038046 22168 ts_tablet_manager.cc:1428] T 1290f70ea9594880b91405e5f04c37c1 P f62d879e7e394f1385bf1a307796c52d: Time spent starting tablet: real 0.037s user 0.023s sys 0.004s
I20260812 07:06:59.039449 22166 ts_tablet_manager.cc:1428] T 1290f70ea9594880b91405e5f04c37c1 P 80498ed6612544e0a5850c817065459d: Time spent starting tablet: real 0.046s user 0.006s sys 0.015s
I20260812 07:06:59.041301 21976 heartbeater.cc:503] Master 127.6.61.62:39677 was elected leader, sending a full tablet report...
W20260812 07:06:59.109542 21978 tablet.cc:2369] T 1290f70ea9594880b91405e5f04c37c1 P 80498ed6612544e0a5850c817065459d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:06:59.146852 22172 raft_consensus.cc:482] T 1290f70ea9594880b91405e5f04c37c1 P 80498ed6612544e0a5850c817065459d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:06:59.147203 22172 raft_consensus.cc:504] T 1290f70ea9594880b91405e5f04c37c1 P 80498ed6612544e0a5850c817065459d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80498ed6612544e0a5850c817065459d" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 33413 } } peers { permanent_uuid: "767e1e4a42774ddb933b6fa833569080" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 41569 } } peers { permanent_uuid: "f62d879e7e394f1385bf1a307796c52d" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42621 } }
I20260812 07:06:59.149323 22172 leader_election.cc:288] T 1290f70ea9594880b91405e5f04c37c1 P 80498ed6612544e0a5850c817065459d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 767e1e4a42774ddb933b6fa833569080 (127.6.61.3:41569), f62d879e7e394f1385bf1a307796c52d (127.6.61.1:42621)
I20260812 07:06:59.169764 22159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:59.150247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:55717 (local address 127.6.61.3:41569)
0812 07:06:59.150490 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:59.150499 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:59.158970 (+ 8471us) server_negotiation.cc:407] Connection header received
0812 07:06:59.159312 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:59.159320 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:59.159407 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:59.159477 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:59.160210 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:59.161680 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:59.167209 (+ 5529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:59.168467 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:59.168564 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:59.168674 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:59.169326 (+ 652us) server_negotiation.cc:299] Negotiation successful
0812 07:06:59.169480 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 07:06:59.170423 22161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:59.159214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:58431 (local address 127.6.61.1:42621)
0812 07:06:59.159431 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:59.159439 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:59.163086 (+ 3647us) server_negotiation.cc:407] Connection header received
0812 07:06:59.163541 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:59.163547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:59.163622 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:59.163679 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:59.164565 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:59.165598 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:59.167995 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:59.169265 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:59.169368 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:59.169459 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:59.170052 (+ 593us) server_negotiation.cc:299] Negotiation successful
0812 07:06:59.170195 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 07:06:59.170847 22175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:59.150415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:41569 (local address 127.6.61.2:55717)
0812 07:06:59.158831 (+ 8416us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:59.158856 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:59.159114 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:59.159697 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:59.159701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:59.159728 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:59.160056 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:59.160071 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:59.161932 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:59.161941 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:59.164397 (+ 2456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:59.164409 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:59.168871 (+ 4462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:59.168879 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:59.169070 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:59.169106 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 07:06:59.170326 (+ 1220us) client_negotiation.cc:240] Negotiation successful
0812 07:06:59.170484 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8225,"thread_start_us":8048,"threads_started":1}
I20260812 07:06:59.172001 22176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:59.159431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:42621 (local address 127.6.61.2:58431)
0812 07:06:59.162962 (+ 3531us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:59.162993 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:59.163359 (+ 366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:59.163885 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:59.163892 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:59.163921 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:59.164358 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:59.164371 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:59.165796 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:59.165801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:59.167776 (+ 1975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:59.167790 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:59.169641 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:59.169647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:59.169850 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:59.169888 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 07:06:59.170823 (+ 935us) client_negotiation.cc:240] Negotiation successful
0812 07:06:59.171607 (+ 784us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3390,"thread_start_us":3664,"threads_started":1}
I20260812 07:06:59.172124 22086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1290f70ea9594880b91405e5f04c37c1" candidate_uuid: "80498ed6612544e0a5850c817065459d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "767e1e4a42774ddb933b6fa833569080" is_pre_election: true
I20260812 07:06:59.172767 22086 raft_consensus.cc:2454] T 1290f70ea9594880b91405e5f04c37c1 P 767e1e4a42774ddb933b6fa833569080 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80498ed6612544e0a5850c817065459d in term 0.
I20260812 07:06:59.173671 21752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1290f70ea9594880b91405e5f04c37c1" candidate_uuid: "80498ed6612544e0a5850c817065459d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f62d879e7e394f1385bf1a307796c52d" is_pre_election: true
I20260812 07:06:59.174561 21752 raft_consensus.cc:2454] T 1290f70ea9594880b91405e5f04c37c1 P f62d879e7e394f1385bf1a307796c52d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80498ed6612544e0a5850c817065459d in term 0.
I20260812 07:06:59.175432 21842 leader_election.cc:302] T 1290f70ea9594880b91405e5f04c37c1 P 80498ed6612544e0a5850c817065459d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80498ed6612544e0a5850c817065459d, f62d879e7e394f1385bf1a307796c52d; no voters:
I20260812 07:06:59.176275 22172 raft_consensus.cc:2781] T 1290f70ea9594880b91405e5f04c37c1 P 80498ed6612544e0a5850c817065459d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:06:59.176420 22172 raft_consensus.cc:482] T 1290f70ea9594880b91405e5f04c37c1 P 80498ed6612544e0a5850c817065459d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:06:59.176507 22172 raft_consensus.cc:3037] T 1290f70ea9594880b91405e5f04c37c1 P 80498ed6612544e0a5850c817065459d [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:59.179880 22172 raft_consensus.cc:504] T 1290f70ea9594880b91405e5f04c37c1 P 80498ed6612544e0a5850c817065459d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80498ed6612544e0a5850c817065459d" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 33413 } } peers { permanent_uuid: "767e1e4a42774ddb933b6fa833569080" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 41569 } } peers { permanent_uuid: "f62d879e7e394f1385bf1a307796c52d" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42621 } }
I20260812 07:06:59.180576 22172 leader_election.cc:288] T 1290f70ea9594880b91405e5f04c37c1 P 80498ed6612544e0a5850c817065459d [CANDIDATE]: Term 1 election: Requested vote from peers 767e1e4a42774ddb933b6fa833569080 (127.6.61.3:41569), f62d879e7e394f1385bf1a307796c52d (127.6.61.1:42621)
I20260812 07:06:59.181402 22086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1290f70ea9594880b91405e5f04c37c1" candidate_uuid: "80498ed6612544e0a5850c817065459d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "767e1e4a42774ddb933b6fa833569080"
I20260812 07:06:59.181461 21752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1290f70ea9594880b91405e5f04c37c1" candidate_uuid: "80498ed6612544e0a5850c817065459d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f62d879e7e394f1385bf1a307796c52d"
I20260812 07:06:59.181633 22086 raft_consensus.cc:3037] T 1290f70ea9594880b91405e5f04c37c1 P 767e1e4a42774ddb933b6fa833569080 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:59.181696 21752 raft_consensus.cc:3037] T 1290f70ea9594880b91405e5f04c37c1 P f62d879e7e394f1385bf1a307796c52d [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:59.185034 21752 raft_consensus.cc:2454] T 1290f70ea9594880b91405e5f04c37c1 P f62d879e7e394f1385bf1a307796c52d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80498ed6612544e0a5850c817065459d in term 1.
I20260812 07:06:59.185034 22086 raft_consensus.cc:2454] T 1290f70ea9594880b91405e5f04c37c1 P 767e1e4a42774ddb933b6fa833569080 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80498ed6612544e0a5850c817065459d in term 1.
I20260812 07:06:59.185634 21842 leader_election.cc:302] T 1290f70ea9594880b91405e5f04c37c1 P 80498ed6612544e0a5850c817065459d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80498ed6612544e0a5850c817065459d, f62d879e7e394f1385bf1a307796c52d; no voters:
I20260812 07:06:59.186226 22172 raft_consensus.cc:2781] T 1290f70ea9594880b91405e5f04c37c1 P 80498ed6612544e0a5850c817065459d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:06:59.191718 22172 raft_consensus.cc:686] T 1290f70ea9594880b91405e5f04c37c1 P 80498ed6612544e0a5850c817065459d [term 1 LEADER]: Becoming Leader. State: Replica: 80498ed6612544e0a5850c817065459d, State: Running, Role: LEADER
I20260812 07:06:59.192445 22172 consensus_queue.cc:237] T 1290f70ea9594880b91405e5f04c37c1 P 80498ed6612544e0a5850c817065459d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80498ed6612544e0a5850c817065459d" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 33413 } } peers { permanent_uuid: "767e1e4a42774ddb933b6fa833569080" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 41569 } } peers { permanent_uuid: "f62d879e7e394f1385bf1a307796c52d" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42621 } }
W20260812 07:06:59.204700 21806 tablet.cc:2369] T 1290f70ea9594880b91405e5f04c37c1 P f62d879e7e394f1385bf1a307796c52d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:06:59.206558 21578 catalog_manager.cc:5495] T 1290f70ea9594880b91405e5f04c37c1 P 80498ed6612544e0a5850c817065459d reported cstate change: term changed from 0 to 1, leader changed from <none> to 80498ed6612544e0a5850c817065459d (127.6.61.2). New cstate: current_term: 1 leader_uuid: "80498ed6612544e0a5850c817065459d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80498ed6612544e0a5850c817065459d" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 33413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "767e1e4a42774ddb933b6fa833569080" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 41569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f62d879e7e394f1385bf1a307796c52d" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42621 } health_report { overall_health: UNKNOWN } } }
I20260812 07:06:59.264997 22158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:59.259190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59646 (local address 127.6.61.2:33413)
0812 07:06:59.259424 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:59.259433 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:59.259556 (+ 123us) server_negotiation.cc:407] Connection header received
0812 07:06:59.259862 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:59.259866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:59.259955 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:59.260115 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:06:59.260536 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:59.261268 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:59.262732 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:59.263152 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:59.263293 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:59.263869 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:06:59.264392 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:06:59.264567 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:59.264626 (+ 59us) server_negotiation.cc:299] Negotiation successful
0812 07:06:59.264720 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 07:06:59.273732 21752 raft_consensus.cc:1261] T 1290f70ea9594880b91405e5f04c37c1 P f62d879e7e394f1385bf1a307796c52d [term 1 FOLLOWER]: Refusing update from remote peer 80498ed6612544e0a5850c817065459d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:06:59.273808 22086 raft_consensus.cc:1261] T 1290f70ea9594880b91405e5f04c37c1 P 767e1e4a42774ddb933b6fa833569080 [term 1 FOLLOWER]: Refusing update from remote peer 80498ed6612544e0a5850c817065459d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:06:59.274772 22177 consensus_queue.cc:1035] T 1290f70ea9594880b91405e5f04c37c1 P 80498ed6612544e0a5850c817065459d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f62d879e7e394f1385bf1a307796c52d" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:06:59.275177 22172 consensus_queue.cc:1035] T 1290f70ea9594880b91405e5f04c37c1 P 80498ed6612544e0a5850c817065459d [LEADER]: Connected to new peer: Peer: permanent_uuid: "767e1e4a42774ddb933b6fa833569080" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 41569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:06:59.294431 22191 mvcc.cc:205] Tried to move back new op lower bound from 7317579445334478848 to 7317579445017403392. Current Snapshot: MvccSnapshot[applied={T|T < 7317579445334478848}]
I20260812 07:06:59.304873 21578 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:06:59.305195 21578 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:06:59.309355 21578 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4b41d60adfc4c0a8bd18700e5e96b52: Sending DeleteTablet for 3 replicas of tablet 1290f70ea9594880b91405e5f04c37c1
I20260812 07:06:59.310629 21897 tablet_service.cc:1514] Processing DeleteTablet for tablet 1290f70ea9594880b91405e5f04c37c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:06:59 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:54410
I20260812 07:06:59.310828 21727 tablet_service.cc:1514] Processing DeleteTablet for tablet 1290f70ea9594880b91405e5f04c37c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:06:59 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:41978
I20260812 07:06:59.311208 22060 tablet_service.cc:1514] Processing DeleteTablet for tablet 1290f70ea9594880b91405e5f04c37c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:06:59 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:37476
I20260812 07:06:59.318013 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 21624
I20260812 07:06:59.326063 22202 tablet_replica.cc:330] T 1290f70ea9594880b91405e5f04c37c1 P f62d879e7e394f1385bf1a307796c52d: stopping tablet replica
I20260812 07:06:59.326833 22201 tablet_replica.cc:330] T 1290f70ea9594880b91405e5f04c37c1 P 80498ed6612544e0a5850c817065459d: stopping tablet replica
I20260812 07:06:59.327435 22201 raft_consensus.cc:2229] T 1290f70ea9594880b91405e5f04c37c1 P 80498ed6612544e0a5850c817065459d [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:06:59.328259 22201 raft_consensus.cc:2258] T 1290f70ea9594880b91405e5f04c37c1 P 80498ed6612544e0a5850c817065459d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:06:59.329101 22203 tablet_replica.cc:330] T 1290f70ea9594880b91405e5f04c37c1 P 767e1e4a42774ddb933b6fa833569080: stopping tablet replica
I20260812 07:06:59.330171 22203 raft_consensus.cc:2229] T 1290f70ea9594880b91405e5f04c37c1 P 767e1e4a42774ddb933b6fa833569080 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:06:59.330130 22202 raft_consensus.cc:2229] T 1290f70ea9594880b91405e5f04c37c1 P f62d879e7e394f1385bf1a307796c52d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:06:59.330698 22202 raft_consensus.cc:2258] T 1290f70ea9594880b91405e5f04c37c1 P f62d879e7e394f1385bf1a307796c52d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:06:59.330849 22201 ts_tablet_manager.cc:1905] T 1290f70ea9594880b91405e5f04c37c1 P 80498ed6612544e0a5850c817065459d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:06:59.332877 22202 ts_tablet_manager.cc:1905] T 1290f70ea9594880b91405e5f04c37c1 P f62d879e7e394f1385bf1a307796c52d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:06:59.333199 22203 raft_consensus.cc:2258] T 1290f70ea9594880b91405e5f04c37c1 P 767e1e4a42774ddb933b6fa833569080 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:06:59.335348 22203 ts_tablet_manager.cc:1905] T 1290f70ea9594880b91405e5f04c37c1 P 767e1e4a42774ddb933b6fa833569080: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:06:59.338697 22201 ts_tablet_manager.cc:1918] T 1290f70ea9594880b91405e5f04c37c1 P 80498ed6612544e0a5850c817065459d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:06:59.338954 22201 log.cc:1196] T 1290f70ea9594880b91405e5f04c37c1 P 80498ed6612544e0a5850c817065459d: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-1/wal/wals/1290f70ea9594880b91405e5f04c37c1
I20260812 07:06:59.339816 22201 ts_tablet_manager.cc:1939] T 1290f70ea9594880b91405e5f04c37c1 P 80498ed6612544e0a5850c817065459d: Deleting consensus metadata
I20260812 07:06:59.340380 22202 ts_tablet_manager.cc:1918] T 1290f70ea9594880b91405e5f04c37c1 P f62d879e7e394f1385bf1a307796c52d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:06:59.340591 22202 log.cc:1196] T 1290f70ea9594880b91405e5f04c37c1 P f62d879e7e394f1385bf1a307796c52d: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-0/wal/wals/1290f70ea9594880b91405e5f04c37c1
I20260812 07:06:59.341133 22202 ts_tablet_manager.cc:1939] T 1290f70ea9594880b91405e5f04c37c1 P f62d879e7e394f1385bf1a307796c52d: Deleting consensus metadata
I20260812 07:06:59.342005 21559 catalog_manager.cc:4841] TS 80498ed6612544e0a5850c817065459d (127.6.61.2:33413): tablet 1290f70ea9594880b91405e5f04c37c1 (table test-table [id=2bf1a3b9ac1444cea5a2ef8de25a3bcf]) successfully deleted
I20260812 07:06:59.343012 21559 catalog_manager.cc:4841] TS f62d879e7e394f1385bf1a307796c52d (127.6.61.1:42621): tablet 1290f70ea9594880b91405e5f04c37c1 (table test-table [id=2bf1a3b9ac1444cea5a2ef8de25a3bcf]) successfully deleted
I20260812 07:06:59.346657 22203 ts_tablet_manager.cc:1918] T 1290f70ea9594880b91405e5f04c37c1 P 767e1e4a42774ddb933b6fa833569080: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:06:59.346860 22203 log.cc:1196] T 1290f70ea9594880b91405e5f04c37c1 P 767e1e4a42774ddb933b6fa833569080: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518208686283-6388-0/minicluster-data/ts-2/wal/wals/1290f70ea9594880b91405e5f04c37c1
I20260812 07:06:59.347381 22203 ts_tablet_manager.cc:1939] T 1290f70ea9594880b91405e5f04c37c1 P 767e1e4a42774ddb933b6fa833569080: Deleting consensus metadata
I20260812 07:06:59.348909 21559 catalog_manager.cc:4841] TS 767e1e4a42774ddb933b6fa833569080 (127.6.61.3:41569): tablet 1290f70ea9594880b91405e5f04c37c1 (table test-table [id=2bf1a3b9ac1444cea5a2ef8de25a3bcf]) successfully deleted
I20260812 07:06:59.352792 22161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:59.319174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41442 (local address 127.6.61.1:42621)
0812 07:06:59.319377 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:59.319384 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:59.329777 (+ 10393us) server_negotiation.cc:407] Connection header received
0812 07:06:59.330559 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:59.330566 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:59.330650 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:59.330787 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:59.335591 (+ 4804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:59.336725 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:59.339539 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:59.340311 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:59.340459 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:59.345325 (+ 4866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:59.345340 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:59.345363 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:59.345416 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:59.349509 (+ 4093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:59.350378 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:59.350389 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:59.350393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:59.350558 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:59.350921 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:59.350927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:59.350931 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:59.351479 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:59.351787 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:59.352081 (+ 294us) server_negotiation.cc:299] Negotiation successful
0812 07:06:59.352378 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 07:06:59.353566 21800 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:06:59.465443 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 21624
I20260812 07:06:59.487332 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 21809
I20260812 07:06:59.503146 22158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:59.488192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59660 (local address 127.6.61.2:33413)
0812 07:06:59.488415 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:59.488422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:59.488474 (+ 52us) server_negotiation.cc:407] Connection header received
0812 07:06:59.488745 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:59.488750 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:59.488820 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:59.488935 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:59.490460 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:59.491293 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:59.492770 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:59.493260 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:59.493418 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:59.496481 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:59.496499 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:59.496519 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:59.496563 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:59.499922 (+ 3359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:59.500796 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:59.500807 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:59.500809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:59.500985 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:59.501413 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:59.501422 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:59.501427 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:59.501793 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:59.502213 (+ 420us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:59.502519 (+ 306us) server_negotiation.cc:299] Negotiation successful
0812 07:06:59.502764 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260812 07:06:59.503867 21971 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:06:59.618577 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 21809
I20260812 07:06:59.642577 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 21981
I20260812 07:06:59.656613 22159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:59.643468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46940 (local address 127.6.61.3:41569)
0812 07:06:59.643735 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:59.643741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:59.643809 (+ 68us) server_negotiation.cc:407] Connection header received
0812 07:06:59.643960 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:59.643964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:59.644031 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:59.644116 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:59.645559 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:59.646380 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:59.647659 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:59.648034 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:59.648126 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:59.651122 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:59.651136 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:59.651154 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:59.651196 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:59.654133 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:59.654732 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:59.654739 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:59.654742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:59.654891 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:59.655250 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:59.655252 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:59.655255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:59.655619 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:59.655850 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:59.656127 (+ 277us) server_negotiation.cc:299] Negotiation successful
0812 07:06:59.656281 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 07:06:59.657415 22131 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:06:59.760411 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 21981
I20260812 07:06:59.782033 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 21536
I20260812 07:06:59.783397 21617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:06:59.897862 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 21536
I20260812 07:06:59.925704 6388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 07:07:00.271703 6388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 07:07:00.271917 6388 mini_ranger.cc:66] Stopping Ranger...
I20260812 07:07:00.617509 6388 mini_ranger.cc:68] Stopped Ranger
2026-08-12 07:07:00.617 UTC [20413] LOG: received smart shutdown request
2026-08-12 07:07:00.621 UTC [20413] LOG: background worker "logical replication launcher" (PID 20424) exited with exit code 1
2026-08-12 07:07:00.621 UTC [20417] LOG: shutting down
2026-08-12 07:07:00.811 UTC [20413] LOG: database system is shut down
2026-08-12T07:07:00Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53953 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:07:01 dist-test-slave-69rg krb5kdc[22218](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:07:01 dist-test-slave-69rg krb5kdc[22218](info): set up 2 sockets
Aug 12 07:07:01 dist-test-slave-69rg krb5kdc[22218](info): commencing operation
krb5kdc: starting...
W20260812 07:07:02.174678 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.181s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:07:02 dist-test-slave-69rg krb5kdc[22218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:07:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:07:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
I20260812 07:07:02.490255 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:35291
--webserver_interface=127.6.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:42827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:35291,127.6.61.61:41341,127.6.61.60:43511
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:07:02.841725 22237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:07:02.842064 22237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:07:02.842159 22237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:07:02.849711 22237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:07:02.849854 22237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:07:02.849903 22237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:07:02.849956 22237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:07:02.850001 22237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:07:02.860141 22237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:42827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:35291,127.6.61.61:41341,127.6.61.60:43511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:35291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:07:02.861490 22237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:07:02.862773 22237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:07:02.875545 22242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:02.875497 22245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:02.875468 22243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:07:02.876973 22237 server_base.cc:1034] running on GCE node
I20260812 07:07:02.877902 22237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:07:02.879454 22237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:07:02.880689 22237 hybrid_clock.cc:648] HybridClock initialized: now 1786518422880641 us; error 39 us; skew 500 ppm
I20260812 07:07:02.881110 22237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:07:02 dist-test-slave-69rg krb5kdc[22218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518422, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:07:02.884517 22237 init.cc:377] Logged in from keytab as kudu/127.6.61.62@KRBTEST.COM (short username kudu)
I20260812 07:07:02.886098 22237 webserver.cc:460] Webserver started at http://127.6.61.62:35691/ using document root <none> and password file <none>
I20260812 07:07:02.887075 22237 fs_manager.cc:359] Metadata directory not provided
I20260812 07:07:02.887224 22237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:07:02.887496 22237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:07:02.890697 22237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "92b0e5da696946cea7fbe2212aaad002"
format_stamp: "Formatted at 2026-08-12 07:07:02 on dist-test-slave-69rg"
server_key: "eedc3b289d3f3d8839454638c859f241"
server_key_iv: "f1019d981a4935b1700267e21fcbd2a3"
server_key_version: "encryptionkey@0"
I20260812 07:07:02.891561 22237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "92b0e5da696946cea7fbe2212aaad002"
format_stamp: "Formatted at 2026-08-12 07:07:02 on dist-test-slave-69rg"
server_key: "eedc3b289d3f3d8839454638c859f241"
server_key_iv: "f1019d981a4935b1700267e21fcbd2a3"
server_key_version: "encryptionkey@0"
I20260812 07:07:02.896981 22237 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260812 07:07:02.900774 22253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:02.902362 22237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 07:07:02.902575 22237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "92b0e5da696946cea7fbe2212aaad002"
format_stamp: "Formatted at 2026-08-12 07:07:02 on dist-test-slave-69rg"
server_key: "eedc3b289d3f3d8839454638c859f241"
server_key_iv: "f1019d981a4935b1700267e21fcbd2a3"
server_key_version: "encryptionkey@0"
I20260812 07:07:02.902796 22237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:07:02.933462 22237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:07:02.938122 22237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:07:02.938457 22237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:07:02.958845 22237 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:35291
I20260812 07:07:02.958904 22310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:35291 every 8 connection(s)
I20260812 07:07:02.960421 22237 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:07:02.961421 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 22237
I20260812 07:07:02.961715 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:07:02.962479 6388 external_mini_cluster.cc:1428] Setting key c4f61102b71517a2136f6c12e273d86b
I20260812 07:07:02.964825 22311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:07:02.972204 22311 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35291 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 07:07:02 dist-test-slave-69rg krb5kdc[22218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518422, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.61.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.61@KRBTEST.COM" created.
I20260812 07:07:02.997996 22314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:02.975865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49420 (local address 127.6.61.62:35291)
0812 07:07:02.976803 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:02.976821 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:02.977317 (+ 496us) server_negotiation.cc:407] Connection header received
0812 07:07:02.980105 (+ 2788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:02.980112 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:02.980392 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:02.980619 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:02.982112 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:02.983420 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:02.984927 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:02.985520 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:02.985748 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:02.989856 (+ 4108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:02.989870 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:02.989891 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:02.989943 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:02.993293 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:02.994271 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:02.994283 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:02.994289 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:02.994452 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:02.994950 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:02.994953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:02.994955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:02.995492 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:02.995682 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:02.996512 (+ 830us) server_negotiation.cc:299] Negotiation successful
0812 07:07:02.996841 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":686,"thread_start_us":451,"threads_started":1}
I20260812 07:07:02.998062 22315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:02.975866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:35291 (local address 127.0.0.1:49420)
0812 07:07:02.977130 (+ 1264us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:02.977167 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:02.979486 (+ 2319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:02.980649 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:02.980656 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:02.981460 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:07:02.981869 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:02.981915 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:02.983602 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:02.983607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:02.984771 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:02.984784 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:02.985804 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:02.985837 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:02.986030 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:02.986769 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:07:02.986794 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:07:02.989505 (+ 2711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:07:02.993446 (+ 3941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:02.993459 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:02.993470 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:02.994106 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:02.994578 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:02.994583 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:02.994586 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:02.994830 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:02.995701 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:07:02.995714 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:07:02.996051 (+ 337us) client_negotiation.cc:769] Sending connection context
0812 07:07:02.996351 (+ 300us) client_negotiation.cc:240] Negotiation successful
0812 07:07:02.997049 (+ 698us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1056,"mutex_wait_us":111,"thread_start_us":838,"threads_started":1}
I20260812 07:07:03.001616 22311 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.61.61" port: 41341 } has no permanent_uuid. Determining permanent_uuid...
I20260812 07:07:03.003095 22315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:03.002495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:41341 (local address 127.0.0.1:59466)
0812 07:07:03.002847 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:03.002973 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.61:41341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260812 07:07:03.003482 22254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.61.61:41341: connect: Connection refused (error 111)
I20260812 07:07:03.005409 22315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:03.005000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:41341 (local address 127.0.0.1:59470)
0812 07:07:03.005202 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:03.005280 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.61:41341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260812 07:07:03.005626 22311 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.61:41341: Network error: Client connection negotiation failed: client connection to 127.6.61.61:41341: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.61.keytab.
Entry for principal kudu/127.6.61.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.61.keytab.
I20260812 07:07:03.031522 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.61:41341
--webserver_interface=127.6.61.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.61.keytab
--principal=kudu/127.6.61.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:42827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:35291,127.6.61.61:41341,127.6.61.60:43511
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:07:03.055018 22311 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.61" port: 41341 } attempt: 1
I20260812 07:07:03.056447 22315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:03.055936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:41341 (local address 127.0.0.1:59486)
0812 07:07:03.056197 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:03.056289 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.61:41341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260812 07:07:03.057641 22315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:03.057240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:41341 (local address 127.0.0.1:59498)
0812 07:07:03.057421 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:03.057495 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.61:41341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260812 07:07:03.057857 22311 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.61:41341: Network error: Client connection negotiation failed: client connection to 127.6.61.61:41341: connect: Connection refused (error 111)
I20260812 07:07:03.126183 22311 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.61" port: 41341 } attempt: 2
I20260812 07:07:03.127535 22315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:03.127023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:41341 (local address 127.0.0.1:59514)
0812 07:07:03.127294 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:03.127391 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.61:41341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260812 07:07:03.128705 22315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:03.128289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:41341 (local address 127.0.0.1:59526)
0812 07:07:03.128472 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:03.128563 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.61:41341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260812 07:07:03.128896 22311 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.61:41341: Network error: Client connection negotiation failed: client connection to 127.6.61.61:41341: connect: Connection refused (error 111)
I20260812 07:07:03.220178 22311 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.61" port: 41341 } attempt: 3
I20260812 07:07:03.221555 22315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:03.221003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:41341 (local address 127.0.0.1:59538)
0812 07:07:03.221296 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:03.221393 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.61:41341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260812 07:07:03.222777 22315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:03.222366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:41341 (local address 127.0.0.1:59554)
0812 07:07:03.222534 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:03.222620 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.61:41341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260812 07:07:03.222973 22311 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.61:41341: Network error: Client connection negotiation failed: client connection to 127.6.61.61:41341: connect: Connection refused (error 111)
I20260812 07:07:03.366266 22311 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.61" port: 41341 } attempt: 4
I20260812 07:07:03.367640 22315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:03.367016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:41341 (local address 127.0.0.1:59568)
0812 07:07:03.367393 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:03.367481 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.61:41341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208}
I20260812 07:07:03.368719 22315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:03.368342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:41341 (local address 127.0.0.1:59580)
0812 07:07:03.368496 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:03.368600 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.61:41341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260812 07:07:03.368898 22311 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.61:41341: Network error: Client connection negotiation failed: client connection to 127.6.61.61:41341: connect: Connection refused (error 111)
W20260812 07:07:03.381309 22318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:07:03.381628 22318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:07:03.381799 22318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:07:03.388979 22318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:07:03.389132 22318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:07:03.389238 22318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:07:03.389348 22318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:07:03.389428 22318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:07:03.399793 22318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:42827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:35291,127.6.61.61:41341,127.6.61.60:43511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.61.keytab
--principal=kudu/127.6.61.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.61:41341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.61.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:07:03.401224 22318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:07:03.402633 22318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:07:03.414734 22324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:03.414829 22325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:03.414788 22327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:07:03.416244 22318 server_base.cc:1034] running on GCE node
I20260812 07:07:03.417078 22318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:07:03.418586 22318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:07:03.419798 22318 hybrid_clock.cc:648] HybridClock initialized: now 1786518423419756 us; error 30 us; skew 500 ppm
I20260812 07:07:03.420168 22318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:07:03 dist-test-slave-69rg krb5kdc[22218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518423, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:07:03.423640 22318 init.cc:377] Logged in from keytab as kudu/127.6.61.61@KRBTEST.COM (short username kudu)
I20260812 07:07:03.425186 22318 webserver.cc:460] Webserver started at http://127.6.61.61:36209/ using document root <none> and password file <none>
I20260812 07:07:03.425967 22318 fs_manager.cc:359] Metadata directory not provided
I20260812 07:07:03.426121 22318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:07:03.426448 22318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:07:03.429518 22318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-1/data/instance:
uuid: "775b84e32866461f8a27459fa2bdd731"
format_stamp: "Formatted at 2026-08-12 07:07:03 on dist-test-slave-69rg"
server_key: "04347f873e778d98bda4b36b838bdba6"
server_key_iv: "aeb5716f7acf4352a9d7843ace40624e"
server_key_version: "encryptionkey@0"
I20260812 07:07:03.430415 22318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-1/wal/instance:
uuid: "775b84e32866461f8a27459fa2bdd731"
format_stamp: "Formatted at 2026-08-12 07:07:03 on dist-test-slave-69rg"
server_key: "04347f873e778d98bda4b36b838bdba6"
server_key_iv: "aeb5716f7acf4352a9d7843ace40624e"
server_key_version: "encryptionkey@0"
I20260812 07:07:03.435674 22318 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 07:07:03.439350 22335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:03.440703 22318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260812 07:07:03.440932 22318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-1/wal
uuid: "775b84e32866461f8a27459fa2bdd731"
format_stamp: "Formatted at 2026-08-12 07:07:03 on dist-test-slave-69rg"
server_key: "04347f873e778d98bda4b36b838bdba6"
server_key_iv: "aeb5716f7acf4352a9d7843ace40624e"
server_key_version: "encryptionkey@0"
I20260812 07:07:03.441154 22318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:07:03.465047 22318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:07:03.469732 22318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:07:03.470090 22318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:07:03.490193 22318 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.61:41341
I20260812 07:07:03.490289 22399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.61:41341 every 8 connection(s)
I20260812 07:07:03.491609 22318 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-1/data/info.pb
I20260812 07:07:03.493481 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 22318
I20260812 07:07:03.493681 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-1/wal/instance
I20260812 07:07:03.494177 6388 external_mini_cluster.cc:1428] Setting key 2e1e55ad145da7b2978e9941a9a1f18c
I20260812 07:07:03.495666 22400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:07:03.502530 22400 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35291 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 07:07:03 dist-test-slave-69rg krb5kdc[22218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518423, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.61@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.61.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.60@KRBTEST.COM" created.
I20260812 07:07:03.524953 22403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:03.505294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49424 (local address 127.6.61.62:35291)
0812 07:07:03.505950 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:03.505957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:03.506929 (+ 972us) server_negotiation.cc:407] Connection header received
0812 07:07:03.508898 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:03.508907 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:03.509007 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:03.509142 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:03.510837 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:03.512138 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:03.513797 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:03.514614 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:03.514761 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:03.518924 (+ 4163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:03.518929 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:03.518932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:03.518977 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:03.521709 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:03.522620 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:03.522624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:03.522626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:03.522770 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:03.523241 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:03.523244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:03.523247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:03.523537 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:03.523660 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:03.524331 (+ 671us) server_negotiation.cc:299] Negotiation successful
0812 07:07:03.524607 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":351,"threads_started":1}
I20260812 07:07:03.526247 22404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:03.505812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:35291 (local address 127.0.0.1:49424)
0812 07:07:03.506753 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:03.506810 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:03.508610 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:03.509419 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:03.509427 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:03.510075 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:07:03.510648 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:03.510665 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:03.512327 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:03.512331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:03.513642 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:03.513656 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:03.514838 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:03.514843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:03.515040 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:03.515790 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:07:03.515818 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:07:03.518669 (+ 2851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:07:03.521894 (+ 3225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:03.521906 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:03.521925 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:03.522473 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:03.522872 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:03.522875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:03.522877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:03.523100 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:03.523648 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:07:03.523653 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:07:03.523977 (+ 324us) client_negotiation.cc:769] Sending connection context
0812 07:07:03.524245 (+ 268us) client_negotiation.cc:240] Negotiation successful
0812 07:07:03.524666 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":425,"threads_started":1}
I20260812 07:07:03.528160 22400 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.61.61" port: 41341 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 07:07:03 dist-test-slave-69rg krb5kdc[22218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518423, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.61@KRBTEST.COM for kudu/127.6.61.61@KRBTEST.COM
I20260812 07:07:03.545161 22404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:03.529059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:41341 (local address 127.0.0.1:59590)
0812 07:07:03.529319 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:03.529341 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:03.529464 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:03.530635 (+ 1171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:03.530638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:03.531005 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:07:03.531153 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:03.531161 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:03.533256 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:03.533260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:03.534213 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:03.534227 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:03.535494 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:03.535499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:03.535658 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:03.536309 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:07:03.536327 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:07:03.538715 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:07:03.541998 (+ 3283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:03.542006 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:03.542013 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:03.542589 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:03.543003 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:03.543012 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:03.543018 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:03.543229 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:03.543876 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:07:03.543883 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:07:03.544110 (+ 227us) client_negotiation.cc:769] Sending connection context
0812 07:07:03.544286 (+ 176us) client_negotiation.cc:240] Negotiation successful
0812 07:07:03.544622 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260812 07:07:03.545184 22405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:03.529177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59590 (local address 127.6.61.61:41341)
0812 07:07:03.530026 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:03.530037 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:03.530072 (+ 35us) server_negotiation.cc:407] Connection header received
0812 07:07:03.530134 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:03.530145 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:03.530453 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:03.530612 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:03.531327 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:03.533032 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:03.534367 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:03.535316 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:03.535465 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:03.538929 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:03.538941 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:03.538957 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:03.538991 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:03.541810 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:03.542742 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:03.542751 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:03.542753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:03.542880 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:03.543346 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:03.543349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:03.543351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:03.543728 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:03.543914 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:03.544506 (+ 592us) server_negotiation.cc:299] Negotiation successful
0812 07:07:03.544759 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":456,"threads_started":1}
I20260812 07:07:03.547880 22400 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.61.60" port: 43511 } has no permanent_uuid. Determining permanent_uuid...
I20260812 07:07:03.549168 22404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:03.548560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:43511 (local address 127.0.0.1:38784)
0812 07:07:03.548788 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:03.548949 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:43511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260812 07:07:03.549463 22341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.61.60:43511: connect: Connection refused (error 111)
I20260812 07:07:03.551353 22404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:03.550874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:43511 (local address 127.0.0.1:38798)
0812 07:07:03.551059 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:03.551183 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:43511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260812 07:07:03.551584 22400 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.60:43511: Network error: Client connection negotiation failed: client connection to 127.6.61.60:43511: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.60.keytab.
Entry for principal kudu/127.6.61.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.60.keytab.
I20260812 07:07:03.562994 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.60:43511
--webserver_interface=127.6.61.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.60.keytab
--principal=kudu/127.6.61.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:42827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:35291,127.6.61.61:41341,127.6.61.60:43511
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:07:03.600970 22400 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.60" port: 43511 } attempt: 1
I20260812 07:07:03.602353 22404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:03.601761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:43511 (local address 127.0.0.1:38808)
0812 07:07:03.602014 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:03.602154 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:43511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260812 07:07:03.603477 22404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:03.603073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:43511 (local address 127.0.0.1:38820)
0812 07:07:03.603238 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:03.603341 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:43511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260812 07:07:03.603716 22400 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.60:43511: Network error: Client connection negotiation failed: client connection to 127.6.61.60:43511: connect: Connection refused (error 111)
I20260812 07:07:03.668161 22311 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.61" port: 41341 } attempt: 5
I20260812 07:07:03.671950 22400 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.60" port: 43511 } attempt: 2
I20260812 07:07:03.672894 22404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:03.672484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:43511 (local address 127.0.0.1:38828)
0812 07:07:03.672654 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:03.672740 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:43511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260812 07:07:03.674116 22404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:03.673677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:43511 (local address 127.0.0.1:38836)
0812 07:07:03.673869 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:03.673958 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:43511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260812 07:07:03.674453 22400 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.60:43511: Network error: Client connection negotiation failed: client connection to 127.6.61.60:43511: connect: Connection refused (error 111)
Aug 12 07:07:03 dist-test-slave-69rg krb5kdc[22218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518422, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for kudu/127.6.61.61@KRBTEST.COM
I20260812 07:07:03.682888 22315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:03.668923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:41341 (local address 127.0.0.1:59596)
0812 07:07:03.669201 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:03.669224 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:03.669366 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:03.669758 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:03.669763 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:03.670360 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:07:03.670595 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:03.670608 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:03.671675 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:03.671680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:03.673063 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:03.673080 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:03.674140 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:03.674146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:03.674375 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:03.675055 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:07:03.675073 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:07:03.677387 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:07:03.680440 (+ 3053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:03.680446 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:03.680450 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:03.680814 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:03.681188 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:03.681193 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:03.681197 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:03.681346 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:03.681908 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:07:03.681913 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:07:03.682079 (+ 166us) client_negotiation.cc:769] Sending connection context
0812 07:07:03.682222 (+ 143us) client_negotiation.cc:240] Negotiation successful
0812 07:07:03.682543 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260812 07:07:03.682888 22405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:03.669102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59596 (local address 127.6.61.61:41341)
0812 07:07:03.669302 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:03.669307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:03.669328 (+ 21us) server_negotiation.cc:407] Connection header received
0812 07:07:03.669503 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:03.669507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:03.669588 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:03.669674 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:03.670784 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:03.671503 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:03.673221 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:03.673980 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:03.674133 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:03.677587 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:03.677592 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:03.677596 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:03.677628 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:03.680289 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:03.680937 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:03.680945 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:03.680948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:03.681072 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:03.681462 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:03.681468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:03.681472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:03.681753 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:03.681948 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:03.682339 (+ 391us) server_negotiation.cc:299] Negotiation successful
0812 07:07:03.682551 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 07:07:03.683921 22311 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.61.60" port: 43511 } has no permanent_uuid. Determining permanent_uuid...
I20260812 07:07:03.684999 22315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:03.684603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:43511 (local address 127.0.0.1:38850)
0812 07:07:03.684783 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:03.684860 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:43511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260812 07:07:03.686034 22315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:03.685675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:43511 (local address 127.0.0.1:38862)
0812 07:07:03.685857 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:03.685933 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:43511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260812 07:07:03.686371 22311 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.60:43511: Network error: Client connection negotiation failed: client connection to 127.6.61.60:43511: connect: Connection refused (error 111)
I20260812 07:07:03.737650 22311 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.60" port: 43511 } attempt: 1
I20260812 07:07:03.738989 22315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:03.738506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:43511 (local address 127.0.0.1:38864)
0812 07:07:03.738719 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:03.738817 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:43511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260812 07:07:03.740021 22315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:03.739683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:43511 (local address 127.0.0.1:38878)
0812 07:07:03.739851 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:03.739918 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:43511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260812 07:07:03.740234 22311 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.60:43511: Network error: Client connection negotiation failed: client connection to 127.6.61.60:43511: connect: Connection refused (error 111)
I20260812 07:07:03.765734 22400 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.60" port: 43511 } attempt: 3
I20260812 07:07:03.767232 22404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:03.766544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:43511 (local address 127.0.0.1:38892)
0812 07:07:03.766850 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:03.766987 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:43511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260812 07:07:03.768623 22404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:03.768048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:43511 (local address 127.0.0.1:38894)
0812 07:07:03.768328 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:03.768444 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:43511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
W20260812 07:07:03.768952 22400 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.60:43511: Network error: Client connection negotiation failed: client connection to 127.6.61.60:43511: connect: Connection refused (error 111)
I20260812 07:07:03.808493 22311 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.60" port: 43511 } attempt: 2
I20260812 07:07:03.809751 22315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:03.809237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:43511 (local address 127.0.0.1:38910)
0812 07:07:03.809483 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:03.809574 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:43511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260812 07:07:03.810774 22315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:03.810431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:43511 (local address 127.0.0.1:38918)
0812 07:07:03.810579 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:03.810661 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:43511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 07:07:03.811012 22311 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.60:43511: Network error: Client connection negotiation failed: client connection to 127.6.61.60:43511: connect: Connection refused (error 111)
W20260812 07:07:03.910820 22409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:07:03.911108 22409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:07:03.911285 22409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260812 07:07:03.912195 22400 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.60" port: 43511 } attempt: 4
I20260812 07:07:03.913394 22404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:03.912841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:43511 (local address 127.0.0.1:38928)
0812 07:07:03.913098 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:03.913203 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:43511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260812 07:07:03.914451 22404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:03.914084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:43511 (local address 127.0.0.1:38944)
0812 07:07:03.914238 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:03.914321 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:43511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260812 07:07:03.914705 22400 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.60:43511: Network error: Client connection negotiation failed: client connection to 127.6.61.60:43511: connect: Connection refused (error 111)
I20260812 07:07:03.917232 22311 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.60" port: 43511 } attempt: 3
I20260812 07:07:03.918180 22315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:03.917742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:43511 (local address 127.0.0.1:38950)
0812 07:07:03.917941 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:03.918025 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:43511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260812 07:07:03.919184 22315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:03.918810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:43511 (local address 127.0.0.1:38964)
0812 07:07:03.919026 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:03.919093 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:43511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260812 07:07:03.919432 22311 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.60:43511: Network error: Client connection negotiation failed: client connection to 127.6.61.60:43511: connect: Connection refused (error 111)
W20260812 07:07:03.919534 22409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:07:03.919687 22409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:07:03.919777 22409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:07:03.919880 22409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:07:03.919948 22409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:07:03.929751 22409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:42827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:35291,127.6.61.61:41341,127.6.61.60:43511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.60.keytab
--principal=kudu/127.6.61.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.60:43511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.61.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:07:03.931149 22409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:07:03.932447 22409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:07:03.944792 22419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:03.945070 22414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:03.945334 22415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:07:03.945680 22409 server_base.cc:1034] running on GCE node
I20260812 07:07:03.946537 22409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:07:03.947999 22409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:07:03.949250 22409 hybrid_clock.cc:648] HybridClock initialized: now 1786518423949204 us; error 51 us; skew 500 ppm
I20260812 07:07:03.949646 22409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:07:03 dist-test-slave-69rg krb5kdc[22218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518423, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:07:03.952934 22409 init.cc:377] Logged in from keytab as kudu/127.6.61.60@KRBTEST.COM (short username kudu)
I20260812 07:07:03.954509 22409 webserver.cc:460] Webserver started at http://127.6.61.60:45293/ using document root <none> and password file <none>
I20260812 07:07:03.955226 22409 fs_manager.cc:359] Metadata directory not provided
I20260812 07:07:03.955389 22409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:07:03.955705 22409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:07:03.958850 22409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-2/data/instance:
uuid: "3a74b010d42d451c99cbf8a7b00b9e17"
format_stamp: "Formatted at 2026-08-12 07:07:03 on dist-test-slave-69rg"
server_key: "cc8ee87f1cfcbef7457e2a830b752bfc"
server_key_iv: "058476e0794250039e1f4ed3ac7b888d"
server_key_version: "encryptionkey@0"
I20260812 07:07:03.959676 22409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-2/wal/instance:
uuid: "3a74b010d42d451c99cbf8a7b00b9e17"
format_stamp: "Formatted at 2026-08-12 07:07:03 on dist-test-slave-69rg"
server_key: "cc8ee87f1cfcbef7457e2a830b752bfc"
server_key_iv: "058476e0794250039e1f4ed3ac7b888d"
server_key_version: "encryptionkey@0"
I20260812 07:07:03.964993 22409 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 07:07:03.968420 22426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:03.969687 22409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 07:07:03.969936 22409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-2/wal
uuid: "3a74b010d42d451c99cbf8a7b00b9e17"
format_stamp: "Formatted at 2026-08-12 07:07:03 on dist-test-slave-69rg"
server_key: "cc8ee87f1cfcbef7457e2a830b752bfc"
server_key_iv: "058476e0794250039e1f4ed3ac7b888d"
server_key_version: "encryptionkey@0"
I20260812 07:07:03.970146 22409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:07:03.997718 22409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:07:04.002338 22409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:07:04.002660 22409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:07:04.022221 22409 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.60:43511
I20260812 07:07:04.022253 22485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.60:43511 every 8 connection(s)
I20260812 07:07:04.023674 22409 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-2/data/info.pb
I20260812 07:07:04.027902 22486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:07:04.032964 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 22409
I20260812 07:07:04.033210 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518208686283-6388-0/minicluster-data/master-2/wal/instance
I20260812 07:07:04.033705 6388 external_mini_cluster.cc:1428] Setting key e6a4c25536d694dd6f5400a9215f01d6
I20260812 07:07:04.035274 22486 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35291 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 07:07:04 dist-test-slave-69rg krb5kdc[22218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518423, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.60@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
Aug 12 07:07:04 dist-test-slave-69rg krb5kdc[22218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.60@KRBTEST.COM
I20260812 07:07:04.058756 22491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:04.038643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49440 (local address 127.6.61.62:35291)
0812 07:07:04.039221 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:04.039229 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:04.039820 (+ 591us) server_negotiation.cc:407] Connection header received
0812 07:07:04.040354 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:04.040360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:04.040439 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:04.040562 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:04.042473 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:04.043615 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:04.045932 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:04.046594 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:04.046735 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:04.051118 (+ 4383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:04.051123 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:04.051127 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:04.051163 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:04.054790 (+ 3627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:04.055700 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:04.055707 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:04.055710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:04.055874 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:04.056461 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:04.056464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:04.056466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:04.056896 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:04.057026 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:04.057988 (+ 962us) server_negotiation.cc:299] Negotiation successful
0812 07:07:04.058287 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":267,"threads_started":1}
I20260812 07:07:04.059868 22490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:04.036062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38966 (local address 127.6.61.60:43511)
0812 07:07:04.037004 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:04.037022 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:04.037079 (+ 57us) server_negotiation.cc:407] Connection header received
0812 07:07:04.039679 (+ 2600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:04.039704 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:04.040069 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:04.040361 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:04.042440 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:04.044084 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:04.046108 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:04.047044 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:04.047139 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:04.051858 (+ 4719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:04.051868 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:04.051881 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:04.051912 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:04.055686 (+ 3774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:04.056599 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:04.056608 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:04.056611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:04.056788 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:04.057323 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:04.057326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:04.057328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:04.057663 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:04.057995 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:04.058433 (+ 438us) server_negotiation.cc:299] Negotiation successful
0812 07:07:04.058684 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":694,"thread_start_us":399,"threads_started":1}
I20260812 07:07:04.059965 22492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:04.038647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:35291 (local address 127.0.0.1:49440)
0812 07:07:04.039649 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:04.039694 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:04.040066 (+ 372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:04.040668 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:04.040677 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:04.041633 (+ 956us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:07:04.042272 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:04.042299 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:04.043803 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:04.043807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:04.045744 (+ 1937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:04.045761 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:04.046840 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:04.046845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:04.047081 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:04.047964 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:07:04.047997 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:07:04.050895 (+ 2898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:07:04.054936 (+ 4041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:04.054946 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:04.054960 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:04.055554 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:04.056001 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:04.056007 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:04.056010 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:04.056333 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:04.057055 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:07:04.057065 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:07:04.057516 (+ 451us) client_negotiation.cc:769] Sending connection context
0812 07:07:04.057807 (+ 291us) client_negotiation.cc:240] Negotiation successful
0812 07:07:04.058473 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"mutex_wait_us":44,"spinlock_wait_cycles":23936,"thread_start_us":563,"threads_started":1}
I20260812 07:07:04.061625 22486 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.61.61" port: 41341 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 07:07:04 dist-test-slave-69rg krb5kdc[22218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518423, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.60@KRBTEST.COM for kudu/127.6.61.61@KRBTEST.COM
Aug 12 07:07:04 dist-test-slave-69rg krb5kdc[22218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:07:04.078413 22405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:04.062645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59600 (local address 127.6.61.61:41341)
0812 07:07:04.063044 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:04.063052 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:04.063332 (+ 280us) server_negotiation.cc:407] Connection header received
0812 07:07:04.063513 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:04.063519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:04.063605 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:04.063729 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:04.064853 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:04.065977 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:04.067452 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:04.068081 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:04.068180 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:04.071839 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:04.071848 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:04.071853 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:04.071919 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:04.075275 (+ 3356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:04.076088 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:04.076095 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:04.076100 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:04.076278 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:04.076835 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:04.076839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:04.076841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:04.077092 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:04.077197 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:04.077937 (+ 740us) server_negotiation.cc:299] Negotiation successful
0812 07:07:04.078120 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229}
I20260812 07:07:04.078567 22492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:04.062585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:41341 (local address 127.0.0.1:59600)
0812 07:07:04.063214 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:04.063236 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:04.063372 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:04.063977 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:04.063985 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:04.064455 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:07:04.064660 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:04.064677 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:04.066218 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:04.066223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:04.067287 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:04.067300 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:04.068234 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:04.068242 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:04.068465 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:04.069092 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:07:04.069116 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:07:04.071609 (+ 2493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:07:04.075381 (+ 3772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:04.075387 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:04.075391 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:04.075935 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:04.076435 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:04.076441 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:04.076446 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:04.076688 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:04.077356 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:07:04.077360 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:07:04.077579 (+ 219us) client_negotiation.cc:769] Sending connection context
0812 07:07:04.077769 (+ 190us) client_negotiation.cc:240] Negotiation successful
0812 07:07:04.078173 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491}
I20260812 07:07:04.079746 22486 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.61.60" port: 43511 } has no permanent_uuid. Determining permanent_uuid...
I20260812 07:07:04.082190 22491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:04.064623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49446 (local address 127.6.61.62:35291)
0812 07:07:04.064847 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:04.064855 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:04.064874 (+ 19us) server_negotiation.cc:407] Connection header received
0812 07:07:04.065083 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:04.065088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:04.065166 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:04.065277 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:04.066886 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:04.067769 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:04.069670 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:04.070137 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:04.070235 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:04.075367 (+ 5132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:04.075373 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:04.075377 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:04.075418 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:04.079222 (+ 3804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:04.079966 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:04.079974 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:04.079978 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:04.080124 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:04.080559 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:04.080572 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:04.080578 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:04.080973 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:04.081295 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:04.081613 (+ 318us) server_negotiation.cc:299] Negotiation successful
0812 07:07:04.081880 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
Aug 12 07:07:04 dist-test-slave-69rg krb5kdc[22218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518423, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.60@KRBTEST.COM for kudu/127.6.61.60@KRBTEST.COM
Aug 12 07:07:04 dist-test-slave-69rg krb5kdc[22218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.61@KRBTEST.COM
I20260812 07:07:04.096033 22492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:04.080454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:43511 (local address 127.0.0.1:38970)
0812 07:07:04.080626 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:04.080645 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:04.080742 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:04.081196 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:04.081201 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:04.081670 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:07:04.081959 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:04.081973 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:04.083289 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:04.083298 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:04.084757 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:04.084774 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:04.085795 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:04.085800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:04.086049 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:04.086623 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:07:04.086649 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:07:04.089245 (+ 2596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:07:04.093327 (+ 4082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:04.093332 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:04.093335 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:04.093967 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:04.094366 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:04.094372 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:04.094376 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:04.094558 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:04.095090 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:07:04.095097 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:07:04.095297 (+ 200us) client_negotiation.cc:769] Sending connection context
0812 07:07:04.095459 (+ 162us) client_negotiation.cc:240] Negotiation successful
0812 07:07:04.095720 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260812 07:07:04.096496 22490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:04.080692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38970 (local address 127.6.61.60:43511)
0812 07:07:04.080885 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:04.080892 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:04.080913 (+ 21us) server_negotiation.cc:407] Connection header received
0812 07:07:04.080964 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:04.080969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:04.081054 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:04.081160 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:04.082128 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:04.083063 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:04.084910 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:04.085627 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:04.085795 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:04.089437 (+ 3642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:04.089442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:04.089445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:04.089474 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:04.093166 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:04.094085 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:04.094090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:04.094092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:04.094234 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:04.094677 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:04.094681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:04.094683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:04.094952 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:04.095061 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:04.095591 (+ 530us) server_negotiation.cc:299] Negotiation successful
0812 07:07:04.095921 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":68}
I20260812 07:07:04.096673 22311 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.60" port: 43511 } attempt: 4
I20260812 07:07:04.102447 22486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a74b010d42d451c99cbf8a7b00b9e17: Bootstrap starting.
I20260812 07:07:04.102856 22405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:04.084147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59602 (local address 127.6.61.61:41341)
0812 07:07:04.084335 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:04.084341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:04.084361 (+ 20us) server_negotiation.cc:407] Connection header received
0812 07:07:04.084445 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:04.084450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:04.084524 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:04.084633 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:04.086216 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:04.087514 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:04.088878 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:04.089570 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:04.089667 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:04.095547 (+ 5880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:04.095554 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:04.095561 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:04.095602 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:04.099529 (+ 3927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:04.100446 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:04.100455 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:04.100458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:04.100653 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:04.101129 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:04.101135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:04.101140 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:04.101516 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:04.101856 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:04.102208 (+ 352us) server_negotiation.cc:299] Negotiation successful
0812 07:07:04.102468 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 07:07:04.106639 22486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a74b010d42d451c99cbf8a7b00b9e17: Neither blocks nor log segments found. Creating new log.
Aug 12 07:07:04 dist-test-slave-69rg krb5kdc[22218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518422, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for kudu/127.6.61.60@KRBTEST.COM
I20260812 07:07:04.108079 22486 log.cc:824] T 00000000000000000000000000000000 P 3a74b010d42d451c99cbf8a7b00b9e17: Log is configured to *not* fsync() on all Append() calls
I20260812 07:07:04.109476 6388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:07:04.111835 22486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a74b010d42d451c99cbf8a7b00b9e17: No bootstrap required, opened a new log
I20260812 07:07:04.116954 22490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:04.097715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38980 (local address 127.6.61.60:43511)
0812 07:07:04.097954 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:04.097962 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:04.097983 (+ 21us) server_negotiation.cc:407] Connection header received
0812 07:07:04.098045 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:04.098050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:04.098151 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:04.099380 (+ 1229us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:04.099540 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:04.100421 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:04.102388 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:04.103639 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:04.104429 (+ 790us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:04.109326 (+ 4897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:04.109330 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:04.109334 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:04.109368 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:04.113080 (+ 3712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:04.114040 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:04.114046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:04.114050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:04.114215 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:04.114759 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:04.114764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:04.114767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:04.115530 (+ 763us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:04.115706 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:04.116306 (+ 600us) server_negotiation.cc:299] Negotiation successful
0812 07:07:04.116566 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 07:07:04.117190 22315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:04.097412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:43511 (local address 127.0.0.1:38980)
0812 07:07:04.097614 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:04.097634 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:04.097746 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:04.098367 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:04.098372 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:04.098914 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:07:04.099184 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:04.099201 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:04.100624 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:04.100630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:04.102158 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:04.102176 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:04.103818 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:04.103827 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:04.104031 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:04.104749 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:07:04.104798 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:07:04.108570 (+ 3772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:07:04.113277 (+ 4707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:04.113283 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:04.113287 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:04.113863 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:04.114391 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:04.114397 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:04.114401 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:04.114601 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:04.115694 (+ 1093us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:07:04.115703 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:07:04.115937 (+ 234us) client_negotiation.cc:769] Sending connection context
0812 07:07:04.116158 (+ 221us) client_negotiation.cc:240] Negotiation successful
0812 07:07:04.116620 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260812 07:07:04.121323 22486 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a74b010d42d451c99cbf8a7b00b9e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0e5da696946cea7fbe2212aaad002" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35291 } } peers { permanent_uuid: "775b84e32866461f8a27459fa2bdd731" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 41341 } } peers { permanent_uuid: "3a74b010d42d451c99cbf8a7b00b9e17" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 43511 } }
I20260812 07:07:04.121721 22486 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a74b010d42d451c99cbf8a7b00b9e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:07:04.121878 22486 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a74b010d42d451c99cbf8a7b00b9e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a74b010d42d451c99cbf8a7b00b9e17, State: Initialized, Role: FOLLOWER
I20260812 07:07:04.122722 22486 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a74b010d42d451c99cbf8a7b00b9e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0e5da696946cea7fbe2212aaad002" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35291 } } peers { permanent_uuid: "775b84e32866461f8a27459fa2bdd731" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 41341 } } peers { permanent_uuid: "3a74b010d42d451c99cbf8a7b00b9e17" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 43511 } }
I20260812 07:07:04.124784 22311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92b0e5da696946cea7fbe2212aaad002: Bootstrap starting.
I20260812 07:07:04.126797 22502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a74b010d42d451c99cbf8a7b00b9e17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0e5da696946cea7fbe2212aaad002" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35291 } } peers { permanent_uuid: "775b84e32866461f8a27459fa2bdd731" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 41341 } } peers { permanent_uuid: "3a74b010d42d451c99cbf8a7b00b9e17" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 43511 } } }
I20260812 07:07:04.127287 22502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a74b010d42d451c99cbf8a7b00b9e17 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:07:04.129343 22311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92b0e5da696946cea7fbe2212aaad002: Neither blocks nor log segments found. Creating new log.
I20260812 07:07:04.129472 22486 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a74b010d42d451c99cbf8a7b00b9e17 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:07:04.130847 22311 log.cc:824] T 00000000000000000000000000000000 P 92b0e5da696946cea7fbe2212aaad002: Log is configured to *not* fsync() on all Append() calls
I20260812 07:07:04.140874 22311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92b0e5da696946cea7fbe2212aaad002: No bootstrap required, opened a new log
I20260812 07:07:04.145156 22405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:04.115316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59614 (local address 127.6.61.61:41341)
0812 07:07:04.115876 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:04.115882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:04.122576 (+ 6694us) server_negotiation.cc:407] Connection header received
0812 07:07:04.122767 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:04.122774 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:04.122866 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:04.122979 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:04.125797 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:04.127050 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:04.129275 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:04.130059 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:04.130178 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:04.135786 (+ 5608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:04.135798 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:04.135806 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:04.135872 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:04.141256 (+ 5384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:04.142263 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:04.142270 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:04.142273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:04.142454 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:04.143020 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:04.143026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:04.143031 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:04.143445 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:04.143688 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:04.144465 (+ 777us) server_negotiation.cc:299] Negotiation successful
0812 07:07:04.144744 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415}
I20260812 07:07:04.154188 22311 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92b0e5da696946cea7fbe2212aaad002 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0e5da696946cea7fbe2212aaad002" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35291 } } peers { permanent_uuid: "775b84e32866461f8a27459fa2bdd731" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 41341 } } peers { permanent_uuid: "3a74b010d42d451c99cbf8a7b00b9e17" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 43511 } }
I20260812 07:07:04.154779 22311 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92b0e5da696946cea7fbe2212aaad002 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:07:04.154929 22311 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92b0e5da696946cea7fbe2212aaad002 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b0e5da696946cea7fbe2212aaad002, State: Initialized, Role: FOLLOWER
I20260812 07:07:04.155881 22311 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92b0e5da696946cea7fbe2212aaad002 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0e5da696946cea7fbe2212aaad002" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35291 } } peers { permanent_uuid: "775b84e32866461f8a27459fa2bdd731" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 41341 } } peers { permanent_uuid: "3a74b010d42d451c99cbf8a7b00b9e17" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 43511 } }
I20260812 07:07:04.159699 22490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:04.131784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38984 (local address 127.6.61.60:43511)
0812 07:07:04.131955 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:04.131961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:04.138795 (+ 6834us) server_negotiation.cc:407] Connection header received
0812 07:07:04.139252 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:04.139271 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:04.139372 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:04.139445 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:04.141635 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:04.143026 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:04.145897 (+ 2871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:04.146742 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:04.146849 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:04.150818 (+ 3969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:04.150827 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:04.150834 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:04.150878 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:04.155178 (+ 4300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:04.156167 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:04.156176 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:04.156181 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:04.156348 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:04.156948 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:04.156957 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:04.156964 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:04.157552 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:04.157932 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:04.158793 (+ 861us) server_negotiation.cc:299] Negotiation successful
0812 07:07:04.159167 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 07:07:04.159889 22491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:04.115290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49450 (local address 127.6.61.62:35291)
0812 07:07:04.115501 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:04.115509 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:04.131589 (+ 16080us) server_negotiation.cc:407] Connection header received
0812 07:07:04.131895 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:04.131901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:04.131984 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:04.132057 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:04.137532 (+ 5475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:04.138930 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:04.143458 (+ 4528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:04.144189 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:04.144299 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:04.149966 (+ 5667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:04.149976 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:04.149984 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:04.150070 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:04.155260 (+ 5190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:04.156566 (+ 1306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:04.156575 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:04.156581 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:04.156769 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:04.157362 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:04.157370 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:04.157377 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:04.157880 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:04.158280 (+ 400us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:04.159021 (+ 741us) server_negotiation.cc:299] Negotiation successful
0812 07:07:04.159330 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 07:07:04.160465 22509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92b0e5da696946cea7fbe2212aaad002 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0e5da696946cea7fbe2212aaad002" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35291 } } peers { permanent_uuid: "775b84e32866461f8a27459fa2bdd731" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 41341 } } peers { permanent_uuid: "3a74b010d42d451c99cbf8a7b00b9e17" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 43511 } } }
I20260812 07:07:04.160895 22509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92b0e5da696946cea7fbe2212aaad002 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:07:04.163512 22311 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92b0e5da696946cea7fbe2212aaad002 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 07:07:04.174912 22520 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3a74b010d42d451c99cbf8a7b00b9e17: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 07:07:04.175053 22520 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 07:07:04.170245 22409 master_runner.cc:186] Error getting master registration for 127.6.61.62:35291: OK, instance_id { permanent_uuid: "92b0e5da696946cea7fbe2212aaad002" instance_seqno: 1786518422952841 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 07:07:04.185401 22409 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260812 07:07:04.191401 22259 tablet.cc:2369] T 00000000000000000000000000000000 P 92b0e5da696946cea7fbe2212aaad002: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 07:07:04.193418 22530 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 92b0e5da696946cea7fbe2212aaad002: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 07:07:04.193670 22530 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 07:07:04.182847 22237 master_runner.cc:186] Error getting master registration for 127.6.61.61:41341: OK, instance_id { permanent_uuid: "775b84e32866461f8a27459fa2bdd731" instance_seqno: 1786518423484561 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 07:07:04.194829 22237 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 07:07:04.214089 22400 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.60" port: 43511 } attempt: 5
Aug 12 07:07:04 dist-test-slave-69rg krb5kdc[22218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518423, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.61@KRBTEST.COM for kudu/127.6.61.60@KRBTEST.COM
I20260812 07:07:04.230733 22404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:04.214673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:43511 (local address 127.0.0.1:38990)
0812 07:07:04.215050 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:04.215073 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:04.215192 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:04.215864 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:04.215868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:04.216443 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:07:04.216665 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:04.216678 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:04.218116 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:04.218123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:04.219008 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:04.219019 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:04.219773 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:04.219779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:04.219969 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:04.220600 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:07:04.220619 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:07:04.222844 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:07:04.227474 (+ 4630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:04.227479 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:04.227483 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:04.228032 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:04.228530 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:04.228534 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:04.228536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:04.228773 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:04.229571 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:07:04.229576 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:07:04.229741 (+ 165us) client_negotiation.cc:769] Sending connection context
0812 07:07:04.229937 (+ 196us) client_negotiation.cc:240] Negotiation successful
0812 07:07:04.230269 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249}
I20260812 07:07:04.230773 22490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:04.215188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38990 (local address 127.6.61.60:43511)
0812 07:07:04.215388 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:04.215399 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:04.215417 (+ 18us) server_negotiation.cc:407] Connection header received
0812 07:07:04.215472 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:04.215477 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:04.215557 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:04.215652 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:04.216856 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:04.217952 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:04.219155 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:04.219610 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:04.219752 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:04.223042 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:04.223047 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:04.223050 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:04.223104 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:04.227308 (+ 4204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:04.228186 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:04.228192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:04.228195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:04.228385 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:04.228921 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:04.228927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:04.228930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:04.229405 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:04.229596 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:04.230056 (+ 460us) server_negotiation.cc:299] Negotiation successful
0812 07:07:04.230322 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 07:07:04.235634 22400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 775b84e32866461f8a27459fa2bdd731: Bootstrap starting.
I20260812 07:07:04.238847 22400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 775b84e32866461f8a27459fa2bdd731: Neither blocks nor log segments found. Creating new log.
I20260812 07:07:04.239980 22400 log.cc:824] T 00000000000000000000000000000000 P 775b84e32866461f8a27459fa2bdd731: Log is configured to *not* fsync() on all Append() calls
I20260812 07:07:04.248319 22400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 775b84e32866461f8a27459fa2bdd731: No bootstrap required, opened a new log
W20260812 07:07:04.254875 22432 tablet.cc:2369] T 00000000000000000000000000000000 P 3a74b010d42d451c99cbf8a7b00b9e17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:07:04.256031 22400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 775b84e32866461f8a27459fa2bdd731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0e5da696946cea7fbe2212aaad002" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35291 } } peers { permanent_uuid: "775b84e32866461f8a27459fa2bdd731" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 41341 } } peers { permanent_uuid: "3a74b010d42d451c99cbf8a7b00b9e17" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 43511 } }
I20260812 07:07:04.256374 22400 raft_consensus.cc:374] T 00000000000000000000000000000000 P 775b84e32866461f8a27459fa2bdd731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:07:04.256521 22400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 775b84e32866461f8a27459fa2bdd731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 775b84e32866461f8a27459fa2bdd731, State: Initialized, Role: FOLLOWER
I20260812 07:07:04.257202 22400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 775b84e32866461f8a27459fa2bdd731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0e5da696946cea7fbe2212aaad002" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35291 } } peers { permanent_uuid: "775b84e32866461f8a27459fa2bdd731" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 41341 } } peers { permanent_uuid: "3a74b010d42d451c99cbf8a7b00b9e17" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 43511 } }
I20260812 07:07:04.262456 22535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 775b84e32866461f8a27459fa2bdd731 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0e5da696946cea7fbe2212aaad002" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35291 } } peers { permanent_uuid: "775b84e32866461f8a27459fa2bdd731" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 41341 } } peers { permanent_uuid: "3a74b010d42d451c99cbf8a7b00b9e17" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 43511 } } }
I20260812 07:07:04.262897 22535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 775b84e32866461f8a27459fa2bdd731 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:07:04.263870 22400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 775b84e32866461f8a27459fa2bdd731 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:07:04.279958 22318 master_runner.cc:263] No leader master found from master 775b84e32866461f8a27459fa2bdd731
I20260812 07:07:04.280119 22318 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260812 07:07:04.283881 22548 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 775b84e32866461f8a27459fa2bdd731: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 07:07:04.284037 22548 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 07:07:04.393263 22502 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a74b010d42d451c99cbf8a7b00b9e17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:07:04.393671 22502 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a74b010d42d451c99cbf8a7b00b9e17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0e5da696946cea7fbe2212aaad002" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35291 } } peers { permanent_uuid: "775b84e32866461f8a27459fa2bdd731" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 41341 } } peers { permanent_uuid: "3a74b010d42d451c99cbf8a7b00b9e17" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 43511 } }
I20260812 07:07:04.395541 22502 leader_election.cc:288] T 00000000000000000000000000000000 P 3a74b010d42d451c99cbf8a7b00b9e17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92b0e5da696946cea7fbe2212aaad002 (127.6.61.62:35291), 775b84e32866461f8a27459fa2bdd731 (127.6.61.61:41341)
I20260812 07:07:04.396831 22282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a74b010d42d451c99cbf8a7b00b9e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92b0e5da696946cea7fbe2212aaad002" is_pre_election: true
I20260812 07:07:04.397518 22282 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 92b0e5da696946cea7fbe2212aaad002 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a74b010d42d451c99cbf8a7b00b9e17 in term 0.
I20260812 07:07:04.398810 22368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a74b010d42d451c99cbf8a7b00b9e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "775b84e32866461f8a27459fa2bdd731" is_pre_election: true
I20260812 07:07:04.399474 22368 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 775b84e32866461f8a27459fa2bdd731 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a74b010d42d451c99cbf8a7b00b9e17 in term 0.
I20260812 07:07:04.400180 22427 leader_election.cc:302] T 00000000000000000000000000000000 P 3a74b010d42d451c99cbf8a7b00b9e17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a74b010d42d451c99cbf8a7b00b9e17, 775b84e32866461f8a27459fa2bdd731; no voters:
I20260812 07:07:04.401209 22502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a74b010d42d451c99cbf8a7b00b9e17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:07:04.401422 22502 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a74b010d42d451c99cbf8a7b00b9e17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:07:04.401553 22502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a74b010d42d451c99cbf8a7b00b9e17 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:07:04.405125 22502 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a74b010d42d451c99cbf8a7b00b9e17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0e5da696946cea7fbe2212aaad002" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35291 } } peers { permanent_uuid: "775b84e32866461f8a27459fa2bdd731" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 41341 } } peers { permanent_uuid: "3a74b010d42d451c99cbf8a7b00b9e17" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 43511 } }
I20260812 07:07:04.406355 22282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a74b010d42d451c99cbf8a7b00b9e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92b0e5da696946cea7fbe2212aaad002"
I20260812 07:07:04.406629 22282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92b0e5da696946cea7fbe2212aaad002 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:07:04.410307 22282 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 92b0e5da696946cea7fbe2212aaad002 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a74b010d42d451c99cbf8a7b00b9e17 in term 1.
I20260812 07:07:04.413134 22430 leader_election.cc:302] T 00000000000000000000000000000000 P 3a74b010d42d451c99cbf8a7b00b9e17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a74b010d42d451c99cbf8a7b00b9e17, 92b0e5da696946cea7fbe2212aaad002; no voters:
I20260812 07:07:04.420948 22368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a74b010d42d451c99cbf8a7b00b9e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "775b84e32866461f8a27459fa2bdd731"
I20260812 07:07:04.421217 22368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 775b84e32866461f8a27459fa2bdd731 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:07:04.424762 22368 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 775b84e32866461f8a27459fa2bdd731 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a74b010d42d451c99cbf8a7b00b9e17 in term 1.
I20260812 07:07:04.425555 22502 leader_election.cc:288] T 00000000000000000000000000000000 P 3a74b010d42d451c99cbf8a7b00b9e17 [CANDIDATE]: Term 1 election: Requested vote from peers 92b0e5da696946cea7fbe2212aaad002 (127.6.61.62:35291), 775b84e32866461f8a27459fa2bdd731 (127.6.61.61:41341)
I20260812 07:07:04.425827 22502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a74b010d42d451c99cbf8a7b00b9e17 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:07:04.426003 22502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a74b010d42d451c99cbf8a7b00b9e17 [term 1 LEADER]: Becoming Leader. State: Replica: 3a74b010d42d451c99cbf8a7b00b9e17, State: Running, Role: LEADER
I20260812 07:07:04.426627 22502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a74b010d42d451c99cbf8a7b00b9e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0e5da696946cea7fbe2212aaad002" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35291 } } peers { permanent_uuid: "775b84e32866461f8a27459fa2bdd731" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 41341 } } peers { permanent_uuid: "3a74b010d42d451c99cbf8a7b00b9e17" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 43511 } }
I20260812 07:07:04.444579 22502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a74b010d42d451c99cbf8a7b00b9e17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a74b010d42d451c99cbf8a7b00b9e17. Latest consensus state: current_term: 1 leader_uuid: "3a74b010d42d451c99cbf8a7b00b9e17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0e5da696946cea7fbe2212aaad002" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35291 } } peers { permanent_uuid: "775b84e32866461f8a27459fa2bdd731" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 41341 } } peers { permanent_uuid: "3a74b010d42d451c99cbf8a7b00b9e17" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 43511 } } }
I20260812 07:07:04.445809 22502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a74b010d42d451c99cbf8a7b00b9e17 [sys.catalog]: This master's current role is: LEADER
I20260812 07:07:04.451165 22554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:07:04.453284 22554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:07:04.462476 22368 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 775b84e32866461f8a27459fa2bdd731 [term 1 FOLLOWER]: Refusing update from remote peer 3a74b010d42d451c99cbf8a7b00b9e17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:07:04.467519 22502 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3a74b010d42d451c99cbf8a7b00b9e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "775b84e32866461f8a27459fa2bdd731" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 41341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 07:07:04.472855 22344 tablet.cc:2369] T 00000000000000000000000000000000 P 775b84e32866461f8a27459fa2bdd731: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:07:04.479816 22282 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 92b0e5da696946cea7fbe2212aaad002 [term 1 FOLLOWER]: Refusing update from remote peer 3a74b010d42d451c99cbf8a7b00b9e17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:07:04.480716 22502 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3a74b010d42d451c99cbf8a7b00b9e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b0e5da696946cea7fbe2212aaad002" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:07:04.506482 22509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92b0e5da696946cea7fbe2212aaad002 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a74b010d42d451c99cbf8a7b00b9e17. Latest consensus state: current_term: 1 leader_uuid: "3a74b010d42d451c99cbf8a7b00b9e17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0e5da696946cea7fbe2212aaad002" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35291 } } peers { permanent_uuid: "775b84e32866461f8a27459fa2bdd731" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 41341 } } peers { permanent_uuid: "3a74b010d42d451c99cbf8a7b00b9e17" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 43511 } } }
I20260812 07:07:04.506887 22509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92b0e5da696946cea7fbe2212aaad002 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:07:04.529279 22555 mvcc.cc:205] Tried to move back new op lower bound from 7317579466585837568 to 7317579466467426304. Current Snapshot: MvccSnapshot[applied={T|T < 7317579466585837568}]
I20260812 07:07:04.535233 22557 mvcc.cc:205] Tried to move back new op lower bound from 7317579466585837568 to 7317579466467426304. Current Snapshot: MvccSnapshot[applied={T|T < 7317579466585837568}]
I20260812 07:07:04.543169 22554 catalog_manager.cc:1347] Generated new cluster ID: cf11dd063ccc42fc8b591edd22ee1cf4
I20260812 07:07:04.543321 22554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:07:04.560230 22535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 775b84e32866461f8a27459fa2bdd731 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a74b010d42d451c99cbf8a7b00b9e17. Latest consensus state: current_term: 1 leader_uuid: "3a74b010d42d451c99cbf8a7b00b9e17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0e5da696946cea7fbe2212aaad002" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35291 } } peers { permanent_uuid: "775b84e32866461f8a27459fa2bdd731" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 41341 } } peers { permanent_uuid: "3a74b010d42d451c99cbf8a7b00b9e17" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 43511 } } }
I20260812 07:07:04.560626 22535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 775b84e32866461f8a27459fa2bdd731 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:07:04.565044 22551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a74b010d42d451c99cbf8a7b00b9e17 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3a74b010d42d451c99cbf8a7b00b9e17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0e5da696946cea7fbe2212aaad002" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35291 } } peers { permanent_uuid: "775b84e32866461f8a27459fa2bdd731" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 41341 } } peers { permanent_uuid: "3a74b010d42d451c99cbf8a7b00b9e17" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 43511 } } }
I20260812 07:07:04.565455 22551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a74b010d42d451c99cbf8a7b00b9e17 [sys.catalog]: This master's current role is: LEADER
I20260812 07:07:04.565467 22535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 775b84e32866461f8a27459fa2bdd731 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3a74b010d42d451c99cbf8a7b00b9e17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0e5da696946cea7fbe2212aaad002" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35291 } } peers { permanent_uuid: "775b84e32866461f8a27459fa2bdd731" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 41341 } } peers { permanent_uuid: "3a74b010d42d451c99cbf8a7b00b9e17" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 43511 } } }
I20260812 07:07:04.565748 22535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 775b84e32866461f8a27459fa2bdd731 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:07:04.566658 22509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92b0e5da696946cea7fbe2212aaad002 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3a74b010d42d451c99cbf8a7b00b9e17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0e5da696946cea7fbe2212aaad002" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35291 } } peers { permanent_uuid: "775b84e32866461f8a27459fa2bdd731" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 41341 } } peers { permanent_uuid: "3a74b010d42d451c99cbf8a7b00b9e17" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 43511 } } }
I20260812 07:07:04.566778 22551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a74b010d42d451c99cbf8a7b00b9e17 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3a74b010d42d451c99cbf8a7b00b9e17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b0e5da696946cea7fbe2212aaad002" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35291 } } peers { permanent_uuid: "775b84e32866461f8a27459fa2bdd731" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 41341 } } peers { permanent_uuid: "3a74b010d42d451c99cbf8a7b00b9e17" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 43511 } } }
I20260812 07:07:04.566938 22509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92b0e5da696946cea7fbe2212aaad002 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:07:04.567022 22551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a74b010d42d451c99cbf8a7b00b9e17 [sys.catalog]: This master's current role is: LEADER
I20260812 07:07:04.574122 22556 mvcc.cc:205] Tried to move back new op lower bound from 7317579466585837568 to 7317579466467426304. Current Snapshot: MvccSnapshot[applied={T|T < 7317579466585837568}]
I20260812 07:07:04.596956 22554 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:07:04.599082 22554 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:07:04.628238 22554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a74b010d42d451c99cbf8a7b00b9e17: Generated new TSK 0
I20260812 07:07:04.629594 22554 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:07:05.195842 22530 catalog_manager.cc:1259] Loaded cluster ID: cf11dd063ccc42fc8b591edd22ee1cf4
I20260812 07:07:05.196008 22530 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 92b0e5da696946cea7fbe2212aaad002: loading cluster ID for follower catalog manager: success
I20260812 07:07:05.198634 22530 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 92b0e5da696946cea7fbe2212aaad002: acquiring CA information for follower catalog manager: success
I20260812 07:07:05.200402 22530 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 92b0e5da696946cea7fbe2212aaad002: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 07:07:05.286198 22548 catalog_manager.cc:1259] Loaded cluster ID: cf11dd063ccc42fc8b591edd22ee1cf4
I20260812 07:07:05.286341 22548 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 775b84e32866461f8a27459fa2bdd731: loading cluster ID for follower catalog manager: success
I20260812 07:07:05.288892 22548 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 775b84e32866461f8a27459fa2bdd731: acquiring CA information for follower catalog manager: success
I20260812 07:07:05.290750 22548 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 775b84e32866461f8a27459fa2bdd731: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 07:07:05.787367 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 22237
I20260812 07:07:05.789263 22305 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:07:05.902205 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 22237
I20260812 07:07:05.928345 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 22318
I20260812 07:07:05.929684 22394 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:07:06.037355 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 22318
I20260812 07:07:06.062911 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 22409
I20260812 07:07:06.064517 22480 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:07:06.175349 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 22409
2026-08-12T07:07:06Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5228 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:07:06 dist-test-slave-69rg krb5kdc[22572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:07:06 dist-test-slave-69rg krb5kdc[22572](info): set up 2 sockets
Aug 12 07:07:06 dist-test-slave-69rg krb5kdc[22572](info): commencing operation
krb5kdc: starting...
W20260812 07:07:07.421963 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.200s user 0.000s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:07:07 dist-test-slave-69rg krb5kdc[22572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:07:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:07:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
I20260812 07:07:07.729441 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:45087
--webserver_interface=127.6.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:38321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:45087
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:07:08.084975 22589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:07:08.085300 22589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:07:08.085392 22589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:07:08.092775 22589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:07:08.092896 22589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:07:08.092943 22589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:07:08.093000 22589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:07:08.093044 22589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:07:08.103554 22589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:38321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:45087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:45087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:07:08.104913 22589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:07:08.106338 22589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:07:08.119078 22595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:08.119136 22598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:08.119164 22594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:07:08.120338 22589 server_base.cc:1034] running on GCE node
I20260812 07:07:08.121228 22589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:07:08.122777 22589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:07:08.123991 22589 hybrid_clock.cc:648] HybridClock initialized: now 1786518428123939 us; error 41 us; skew 500 ppm
I20260812 07:07:08.124366 22589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:07:08 dist-test-slave-69rg krb5kdc[22572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518428, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:07:08.127681 22589 init.cc:377] Logged in from keytab as kudu/127.6.61.62@KRBTEST.COM (short username kudu)
I20260812 07:07:08.129264 22589 webserver.cc:460] Webserver started at http://127.6.61.62:44293/ using document root <none> and password file <none>
I20260812 07:07:08.130004 22589 fs_manager.cc:359] Metadata directory not provided
I20260812 07:07:08.130112 22589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:07:08.130388 22589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:07:08.133358 22589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "459655fa5b9e4dd89be26d5c16284f28"
format_stamp: "Formatted at 2026-08-12 07:07:08 on dist-test-slave-69rg"
server_key: "1bb2b6b1e14e720110b126be9e768154"
server_key_iv: "aa2393d543bc94d873d46be446a1172a"
server_key_version: "encryptionkey@0"
I20260812 07:07:08.134169 22589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "459655fa5b9e4dd89be26d5c16284f28"
format_stamp: "Formatted at 2026-08-12 07:07:08 on dist-test-slave-69rg"
server_key: "1bb2b6b1e14e720110b126be9e768154"
server_key_iv: "aa2393d543bc94d873d46be446a1172a"
server_key_version: "encryptionkey@0"
I20260812 07:07:08.140218 22589 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 07:07:08.143900 22608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:08.145311 22589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 07:07:08.145470 22589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "459655fa5b9e4dd89be26d5c16284f28"
format_stamp: "Formatted at 2026-08-12 07:07:08 on dist-test-slave-69rg"
server_key: "1bb2b6b1e14e720110b126be9e768154"
server_key_iv: "aa2393d543bc94d873d46be446a1172a"
server_key_version: "encryptionkey@0"
I20260812 07:07:08.145656 22589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:07:08.180423 22589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:07:08.184957 22589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:07:08.185309 22589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:07:08.205024 22589 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:45087
I20260812 07:07:08.205063 22665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:45087 every 8 connection(s)
I20260812 07:07:08.206535 22589 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:07:08.210729 22666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:07:08.210793 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 22589
I20260812 07:07:08.211020 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:07:08.211968 6388 external_mini_cluster.cc:1428] Setting key 31989c9bcb64582b3a9b0c94b45cab7e
I20260812 07:07:08.225432 22666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 459655fa5b9e4dd89be26d5c16284f28: Bootstrap starting.
Aug 12 07:07:08 dist-test-slave-69rg krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:07:08.229683 22666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 459655fa5b9e4dd89be26d5c16284f28: Neither blocks nor log segments found. Creating new log.
I20260812 07:07:08.231216 22666 log.cc:824] T 00000000000000000000000000000000 P 459655fa5b9e4dd89be26d5c16284f28: Log is configured to *not* fsync() on all Append() calls
I20260812 07:07:08.235100 22666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 459655fa5b9e4dd89be26d5c16284f28: No bootstrap required, opened a new log
I20260812 07:07:08.239944 22671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:08.213986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33580 (local address 127.6.61.62:45087)
0812 07:07:08.214880 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:08.214892 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:08.214933 (+ 41us) server_negotiation.cc:407] Connection header received
0812 07:07:08.217151 (+ 2218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:08.217176 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:08.217567 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:08.218010 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:08.219927 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:08.221355 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:08.223416 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:08.224167 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:08.224315 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:08.230089 (+ 5774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:08.230104 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:08.230133 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:08.230205 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:08.234570 (+ 4365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:08.235583 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:08.235606 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:08.235618 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:08.235853 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:08.236340 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:08.236345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:08.236348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:08.237177 (+ 829us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:08.237501 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:08.237975 (+ 474us) server_negotiation.cc:299] Negotiation successful
0812 07:07:08.238463 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":423,"threads_started":1}
I20260812 07:07:08.244261 6388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:07:08.245215 22666 raft_consensus.cc:348] T 00000000000000000000000000000000 P 459655fa5b9e4dd89be26d5c16284f28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459655fa5b9e4dd89be26d5c16284f28" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 45087 } }
I20260812 07:07:08.245673 22666 raft_consensus.cc:374] T 00000000000000000000000000000000 P 459655fa5b9e4dd89be26d5c16284f28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:07:08.245792 22666 raft_consensus.cc:729] T 00000000000000000000000000000000 P 459655fa5b9e4dd89be26d5c16284f28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 459655fa5b9e4dd89be26d5c16284f28, State: Initialized, Role: FOLLOWER
I20260812 07:07:08.246507 22666 consensus_queue.cc:260] T 00000000000000000000000000000000 P 459655fa5b9e4dd89be26d5c16284f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459655fa5b9e4dd89be26d5c16284f28" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 45087 } }
I20260812 07:07:08.246802 22666 raft_consensus.cc:388] T 00000000000000000000000000000000 P 459655fa5b9e4dd89be26d5c16284f28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:07:08.246949 22666 raft_consensus.cc:482] T 00000000000000000000000000000000 P 459655fa5b9e4dd89be26d5c16284f28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:07:08.247167 22666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 459655fa5b9e4dd89be26d5c16284f28 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:07:08.249989 22666 raft_consensus.cc:504] T 00000000000000000000000000000000 P 459655fa5b9e4dd89be26d5c16284f28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459655fa5b9e4dd89be26d5c16284f28" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 45087 } }
I20260812 07:07:08.250480 22666 leader_election.cc:302] T 00000000000000000000000000000000 P 459655fa5b9e4dd89be26d5c16284f28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 459655fa5b9e4dd89be26d5c16284f28; no voters:
I20260812 07:07:08.250998 22666 leader_election.cc:288] T 00000000000000000000000000000000 P 459655fa5b9e4dd89be26d5c16284f28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:07:08.251171 22673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 459655fa5b9e4dd89be26d5c16284f28 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:07:08.251889 22673 raft_consensus.cc:686] T 00000000000000000000000000000000 P 459655fa5b9e4dd89be26d5c16284f28 [term 1 LEADER]: Becoming Leader. State: Replica: 459655fa5b9e4dd89be26d5c16284f28, State: Running, Role: LEADER
I20260812 07:07:08.252398 22673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 459655fa5b9e4dd89be26d5c16284f28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459655fa5b9e4dd89be26d5c16284f28" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 45087 } }
I20260812 07:07:08.253165 22666 sys_catalog.cc:564] T 00000000000000000000000000000000 P 459655fa5b9e4dd89be26d5c16284f28 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:07:08.255649 22675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 459655fa5b9e4dd89be26d5c16284f28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 459655fa5b9e4dd89be26d5c16284f28. Latest consensus state: current_term: 1 leader_uuid: "459655fa5b9e4dd89be26d5c16284f28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459655fa5b9e4dd89be26d5c16284f28" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 45087 } } }
I20260812 07:07:08.256158 22675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 459655fa5b9e4dd89be26d5c16284f28 [sys.catalog]: This master's current role is: LEADER
I20260812 07:07:08.255733 22674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 459655fa5b9e4dd89be26d5c16284f28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "459655fa5b9e4dd89be26d5c16284f28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459655fa5b9e4dd89be26d5c16284f28" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 45087 } } }
I20260812 07:07:08.256460 22674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 459655fa5b9e4dd89be26d5c16284f28 [sys.catalog]: This master's current role is: LEADER
I20260812 07:07:08.258244 22685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:07:08.262708 22685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:07:08.271204 22685 catalog_manager.cc:1347] Generated new cluster ID: 166e5c3467944387a3280afda326a01a
I20260812 07:07:08.271332 22685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:07:08.297799 22685 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:07:08.299471 22685 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:07:08.306005 22685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 459655fa5b9e4dd89be26d5c16284f28: Generated new TSK 0
I20260812 07:07:08.306936 22685 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:07:08.359167 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 22589
I20260812 07:07:08.360462 22661 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:07:08.460789 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 22589
2026-08-12T07:07:08Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2278 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:07:08 dist-test-slave-69rg krb5kdc[22702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:07:08 dist-test-slave-69rg krb5kdc[22702](info): set up 2 sockets
Aug 12 07:07:08 dist-test-slave-69rg krb5kdc[22702](info): commencing operation
krb5kdc: starting...
W20260812 07:07:09.679682 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.179s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:07:09 dist-test-slave-69rg krb5kdc[22702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:07:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:07:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
I20260812 07:07:09.990592 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:43861
--webserver_interface=127.6.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:45733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:43861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:07:10.344611 22719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:07:10.344916 22719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:07:10.345077 22719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:07:10.352445 22719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:07:10.352552 22719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:07:10.352656 22719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:07:10.352737 22719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:07:10.352813 22719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:07:10.363065 22719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:45733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:43861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:43861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:07:10.364498 22719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:07:10.365933 22719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:07:10.378497 22725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:10.378517 22728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:10.378558 22726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:07:10.379922 22719 server_base.cc:1034] running on GCE node
I20260812 07:07:10.380987 22719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:07:10.382710 22719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:07:10.383970 22719 hybrid_clock.cc:648] HybridClock initialized: now 1786518430383887 us; error 56 us; skew 500 ppm
I20260812 07:07:10.384451 22719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:07:10 dist-test-slave-69rg krb5kdc[22702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518430, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:07:10.393600 22719 init.cc:377] Logged in from keytab as kudu/127.6.61.62@KRBTEST.COM (short username kudu)
I20260812 07:07:10.395577 22719 webserver.cc:460] Webserver started at http://127.6.61.62:36017/ using document root <none> and password file <none>
I20260812 07:07:10.396382 22719 fs_manager.cc:359] Metadata directory not provided
I20260812 07:07:10.396531 22719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:07:10.396899 22719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:07:10.400614 22719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "de11599be8e84615bc177368a6e7fb04"
format_stamp: "Formatted at 2026-08-12 07:07:10 on dist-test-slave-69rg"
server_key: "c35c551071f6d503fabb514bea3aaba4"
server_key_iv: "6d74cb59effa8d5cf2cf2841f2d1c9f5"
server_key_version: "encryptionkey@0"
I20260812 07:07:10.401616 22719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "de11599be8e84615bc177368a6e7fb04"
format_stamp: "Formatted at 2026-08-12 07:07:10 on dist-test-slave-69rg"
server_key: "c35c551071f6d503fabb514bea3aaba4"
server_key_iv: "6d74cb59effa8d5cf2cf2841f2d1c9f5"
server_key_version: "encryptionkey@0"
I20260812 07:07:10.407620 22719 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260812 07:07:10.411648 22735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:10.413133 22719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 07:07:10.413358 22719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "de11599be8e84615bc177368a6e7fb04"
format_stamp: "Formatted at 2026-08-12 07:07:10 on dist-test-slave-69rg"
server_key: "c35c551071f6d503fabb514bea3aaba4"
server_key_iv: "6d74cb59effa8d5cf2cf2841f2d1c9f5"
server_key_version: "encryptionkey@0"
I20260812 07:07:10.413561 22719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:07:10.430540 22719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:07:10.434914 22719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:07:10.435173 22719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:07:10.454658 22719 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:43861
I20260812 07:07:10.454708 22793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:43861 every 8 connection(s)
I20260812 07:07:10.456074 22719 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:07:10.459882 22795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:07:10.463173 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 22719
I20260812 07:07:10.463421 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:07:10.464306 6388 external_mini_cluster.cc:1428] Setting key e9767f3a5bdcff29d0917b61c010818e
I20260812 07:07:10.473536 22795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04: Bootstrap starting.
I20260812 07:07:10.477445 22795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04: Neither blocks nor log segments found. Creating new log.
I20260812 07:07:10.478878 22795 log.cc:824] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04: Log is configured to *not* fsync() on all Append() calls
Aug 12 07:07:10 dist-test-slave-69rg krb5kdc[22702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:07:10.482344 22795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04: No bootstrap required, opened a new log
I20260812 07:07:10.489705 22798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:10.466665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49910 (local address 127.6.61.62:43861)
0812 07:07:10.467505 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:10.467520 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:10.467570 (+ 50us) server_negotiation.cc:407] Connection header received
0812 07:07:10.469259 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:10.469283 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:10.469680 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:10.470080 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:10.471930 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:10.473253 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:10.475107 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:10.475791 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:10.475919 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:10.481389 (+ 5470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:10.481397 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:10.481411 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:10.481450 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:10.484815 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:10.485730 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:10.485742 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:10.485749 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:10.486032 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:10.486535 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:10.486539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:10.486541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:10.487194 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:10.487438 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:10.487953 (+ 515us) server_negotiation.cc:299] Negotiation successful
0812 07:07:10.488470 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":392,"threads_started":1}
I20260812 07:07:10.491991 22795 raft_consensus.cc:348] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de11599be8e84615bc177368a6e7fb04" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 43861 } }
I20260812 07:07:10.492385 22795 raft_consensus.cc:374] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:07:10.492515 22795 raft_consensus.cc:729] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de11599be8e84615bc177368a6e7fb04, State: Initialized, Role: FOLLOWER
I20260812 07:07:10.493196 22795 consensus_queue.cc:260] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de11599be8e84615bc177368a6e7fb04" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 43861 } }
I20260812 07:07:10.493445 22795 raft_consensus.cc:388] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:07:10.493599 22795 raft_consensus.cc:482] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:07:10.493769 22795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:07:10.496886 22795 raft_consensus.cc:504] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de11599be8e84615bc177368a6e7fb04" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 43861 } }
I20260812 07:07:10.497391 22795 leader_election.cc:302] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de11599be8e84615bc177368a6e7fb04; no voters:
I20260812 07:07:10.498088 22795 leader_election.cc:288] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:07:10.498211 22800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:07:10.498957 22800 raft_consensus.cc:686] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [term 1 LEADER]: Becoming Leader. State: Replica: de11599be8e84615bc177368a6e7fb04, State: Running, Role: LEADER
I20260812 07:07:10.499544 22800 consensus_queue.cc:237] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de11599be8e84615bc177368a6e7fb04" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 43861 } }
I20260812 07:07:10.500326 22795 sys_catalog.cc:564] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:07:10.502722 22803 sys_catalog.cc:455] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de11599be8e84615bc177368a6e7fb04. Latest consensus state: current_term: 1 leader_uuid: "de11599be8e84615bc177368a6e7fb04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de11599be8e84615bc177368a6e7fb04" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 43861 } } }
I20260812 07:07:10.503253 22803 sys_catalog.cc:458] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [sys.catalog]: This master's current role is: LEADER
I20260812 07:07:10.502842 22802 sys_catalog.cc:455] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de11599be8e84615bc177368a6e7fb04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de11599be8e84615bc177368a6e7fb04" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 43861 } } }
I20260812 07:07:10.503435 22802 sys_catalog.cc:458] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [sys.catalog]: This master's current role is: LEADER
I20260812 07:07:10.510815 22807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:07:10.517169 22807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:07:10.529243 22807 catalog_manager.cc:1347] Generated new cluster ID: 137db745a3ca4d959d8b8dc6c9f5b847
I20260812 07:07:10.529390 22807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:07:10.561765 22807 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:07:10.564216 22807 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:07:10.573060 22807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04: Generated new TSK 0
I20260812 07:07:10.574167 22807 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
I20260812 07:07:10.649507 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.1:0
--local_ip_for_outbound_sockets=127.6.61.1
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:43861
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:45733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:07:11.022261 22826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:07:11.022647 22826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:07:11.022814 22826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:07:11.030198 22826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:07:11.030359 22826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:07:11.030550 22826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.1
I20260812 07:07:11.040457 22826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:45733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:43861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.1
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:07:11.041947 22826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:07:11.043337 22826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:07:11.057403 22832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:11.057547 22833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:11.057583 22836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:07:11.058714 22826 server_base.cc:1034] running on GCE node
I20260812 07:07:11.059418 22826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:07:11.060571 22826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:07:11.062002 22826 hybrid_clock.cc:648] HybridClock initialized: now 1786518431061948 us; error 35 us; skew 500 ppm
I20260812 07:07:11.062374 22826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:07:11 dist-test-slave-69rg krb5kdc[22702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518431, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:07:11.066247 22826 init.cc:377] Logged in from keytab as kudu/127.6.61.1@KRBTEST.COM (short username kudu)
I20260812 07:07:11.067891 22826 webserver.cc:460] Webserver started at http://127.6.61.1:46173/ using document root <none> and password file <none>
I20260812 07:07:11.068674 22826 fs_manager.cc:359] Metadata directory not provided
I20260812 07:07:11.068835 22826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:07:11.069228 22826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:07:11.072381 22826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/data/instance:
uuid: "c01e069207904975a3e673740a311e08"
format_stamp: "Formatted at 2026-08-12 07:07:11 on dist-test-slave-69rg"
server_key: "aec759c06f64446dd7d3082a926807da"
server_key_iv: "58116310d115f420baf254d11a6a550b"
server_key_version: "encryptionkey@0"
I20260812 07:07:11.073185 22826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance:
uuid: "c01e069207904975a3e673740a311e08"
format_stamp: "Formatted at 2026-08-12 07:07:11 on dist-test-slave-69rg"
server_key: "aec759c06f64446dd7d3082a926807da"
server_key_iv: "58116310d115f420baf254d11a6a550b"
server_key_version: "encryptionkey@0"
I20260812 07:07:11.078794 22826 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260812 07:07:11.082865 22843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:11.084326 22826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260812 07:07:11.084522 22826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/wal
uuid: "c01e069207904975a3e673740a311e08"
format_stamp: "Formatted at 2026-08-12 07:07:11 on dist-test-slave-69rg"
server_key: "aec759c06f64446dd7d3082a926807da"
server_key_iv: "58116310d115f420baf254d11a6a550b"
server_key_version: "encryptionkey@0"
I20260812 07:07:11.084748 22826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:07:11.110728 22826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:07:11.115213 22826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:07:11.115532 22826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:07:11.116533 22826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:07:11.118073 22826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:07:11.118198 22826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:11.118358 22826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:07:11.118430 22826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:11.154170 22826 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.1:36915
I20260812 07:07:11.154263 22971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.1:36915 every 8 connection(s)
I20260812 07:07:11.155545 22826 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
I20260812 07:07:11.165392 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 22826
I20260812 07:07:11.165634 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance
I20260812 07:07:11.166374 6388 external_mini_cluster.cc:1428] Setting key 84ed73ea454e6e47fdf92200b8422df0
Aug 12 07:07:11 dist-test-slave-69rg krb5kdc[22702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518431, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:07:11.177186 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:11.158314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:36575 (local address 127.6.61.62:43861)
0812 07:07:11.158945 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:11.158953 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:11.159925 (+ 972us) server_negotiation.cc:407] Connection header received
0812 07:07:11.161785 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:11.161790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:11.161910 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:11.162070 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:11.163578 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:11.164321 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:11.166021 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:11.166433 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:11.166578 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:11.170704 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:11.170709 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:11.170713 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:11.170747 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:11.173540 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:11.174641 (+ 1101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:11.174650 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:11.174654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:11.174802 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:11.175313 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:11.175318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:11.175320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:11.175604 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:11.175776 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:11.176558 (+ 782us) server_negotiation.cc:299] Negotiation successful
0812 07:07:11.176808 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":296,"threads_started":1}
I20260812 07:07:11.178339 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:11.158936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.1:36575)
0812 07:07:11.159753 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:11.159790 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:11.161506 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:11.162336 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:11.162345 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:11.162850 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:07:11.163406 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:11.163424 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:11.164487 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:11.164492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:11.165869 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:11.165884 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:11.166692 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:11.166700 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:11.166995 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:11.167879 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:07:11.167907 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:07:11.170453 (+ 2546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:07:11.174051 (+ 3598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:11.174058 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:11.174074 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:11.174510 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:11.174936 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:11.174945 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:11.174948 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:11.175169 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:11.175762 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:07:11.175772 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:07:11.176134 (+ 362us) client_negotiation.cc:769] Sending connection context
0812 07:07:11.176429 (+ 295us) client_negotiation.cc:240] Negotiation successful
0812 07:07:11.177007 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":354,"threads_started":1}
I20260812 07:07:11.180430 22972 heartbeater.cc:347] Connected to a master server at 127.6.61.62:43861
I20260812 07:07:11.180792 22972 heartbeater.cc:464] Registering TS with master...
I20260812 07:07:11.181547 22972 heartbeater.cc:511] Master 127.6.61.62:43861 requested a full tablet report, sending...
I20260812 07:07:11.183821 22755 ts_manager.cc:194] Registered new tserver with Master: c01e069207904975a3e673740a311e08 (127.6.61.1:36915)
I20260812 07:07:11.185309 22755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.1@KRBTEST.COM'} at 127.6.61.1:36575
WARNING: no policy specified for kudu/127.6.61.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
I20260812 07:07:11.237208 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.2:0
--local_ip_for_outbound_sockets=127.6.61.2
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:43861
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:45733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:07:11.583860 22979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:07:11.584156 22979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:07:11.584319 22979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:07:11.591893 22979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:07:11.592017 22979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:07:11.592236 22979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.2
I20260812 07:07:11.602336 22979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:45733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:43861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.2
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:07:11.603729 22979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:07:11.605093 22979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:07:11.618803 22987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:11.618757 22990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:11.618849 22988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:07:11.620329 22979 server_base.cc:1034] running on GCE node
I20260812 07:07:11.620965 22979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:07:11.621989 22979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:07:11.623227 22979 hybrid_clock.cc:648] HybridClock initialized: now 1786518431623190 us; error 47 us; skew 500 ppm
I20260812 07:07:11.623725 22979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:07:11 dist-test-slave-69rg krb5kdc[22702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518431, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:07:11.631786 22979 init.cc:377] Logged in from keytab as kudu/127.6.61.2@KRBTEST.COM (short username kudu)
I20260812 07:07:11.633430 22979 webserver.cc:460] Webserver started at http://127.6.61.2:39753/ using document root <none> and password file <none>
I20260812 07:07:11.634269 22979 fs_manager.cc:359] Metadata directory not provided
I20260812 07:07:11.634428 22979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:07:11.634748 22979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:07:11.637929 22979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/data/instance:
uuid: "74c1eb79d71c44a5b1de64128768e0b8"
format_stamp: "Formatted at 2026-08-12 07:07:11 on dist-test-slave-69rg"
server_key: "00bcc27a935bc7bc906722eeeb848a12"
server_key_iv: "b5df1704f6660f6017915cf24c0b7c27"
server_key_version: "encryptionkey@0"
I20260812 07:07:11.638803 22979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance:
uuid: "74c1eb79d71c44a5b1de64128768e0b8"
format_stamp: "Formatted at 2026-08-12 07:07:11 on dist-test-slave-69rg"
server_key: "00bcc27a935bc7bc906722eeeb848a12"
server_key_iv: "b5df1704f6660f6017915cf24c0b7c27"
server_key_version: "encryptionkey@0"
I20260812 07:07:11.644316 22979 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 07:07:11.647764 22998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:11.649047 22979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 07:07:11.649235 22979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/wal
uuid: "74c1eb79d71c44a5b1de64128768e0b8"
format_stamp: "Formatted at 2026-08-12 07:07:11 on dist-test-slave-69rg"
server_key: "00bcc27a935bc7bc906722eeeb848a12"
server_key_iv: "b5df1704f6660f6017915cf24c0b7c27"
server_key_version: "encryptionkey@0"
I20260812 07:07:11.649451 22979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:07:11.674882 22979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:07:11.679652 22979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:07:11.679981 22979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:07:11.681017 22979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:07:11.682562 22979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:07:11.682672 22979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:11.682823 22979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:07:11.682898 22979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:11.718116 22979 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.2:42331
I20260812 07:07:11.718150 23130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.2:42331 every 8 connection(s)
I20260812 07:07:11.719478 22979 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
I20260812 07:07:11.728119 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 22979
I20260812 07:07:11.728338 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance
I20260812 07:07:11.728824 6388 external_mini_cluster.cc:1428] Setting key 2a96e850b971ed96ba4d08c4c1aea038
Aug 12 07:07:11 dist-test-slave-69rg krb5kdc[22702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518431, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:07:11.740917 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:11.722313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:56543 (local address 127.6.61.62:43861)
0812 07:07:11.722821 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:11.722828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:11.723888 (+ 1060us) server_negotiation.cc:407] Connection header received
0812 07:07:11.725782 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:11.725787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:11.725928 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:11.726029 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:11.727592 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:11.728371 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:11.730216 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:11.730810 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:11.730997 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:11.734900 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:11.734906 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:11.734909 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:11.734945 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:11.737676 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:11.738495 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:11.738501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:11.738505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:11.738651 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:11.739119 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:11.739123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:11.739125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:11.739393 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:11.739560 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:11.740283 (+ 723us) server_negotiation.cc:299] Negotiation successful
0812 07:07:11.740521 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":223,"threads_started":1}
I20260812 07:07:11.741977 23138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:11.722875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.2:56543)
0812 07:07:11.723697 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:11.723738 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:11.725498 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:11.726243 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:11.726252 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:11.726813 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:07:11.727417 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:11.727435 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:11.728549 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:11.728554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:11.730061 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:11.730074 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:11.731089 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:11.731097 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:11.731341 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:11.732077 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:07:11.732100 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:07:11.734621 (+ 2521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:07:11.737866 (+ 3245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:11.737875 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:11.737894 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:11.738362 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:11.738769 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:11.738777 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:11.738781 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:11.738981 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:11.739539 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:07:11.739546 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:07:11.739898 (+ 352us) client_negotiation.cc:769] Sending connection context
0812 07:07:11.740168 (+ 270us) client_negotiation.cc:240] Negotiation successful
0812 07:07:11.740726 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":341,"threads_started":1}
I20260812 07:07:11.743548 23133 heartbeater.cc:347] Connected to a master server at 127.6.61.62:43861
I20260812 07:07:11.743932 23133 heartbeater.cc:464] Registering TS with master...
I20260812 07:07:11.744827 23133 heartbeater.cc:511] Master 127.6.61.62:43861 requested a full tablet report, sending...
I20260812 07:07:11.746798 22755 ts_manager.cc:194] Registered new tserver with Master: 74c1eb79d71c44a5b1de64128768e0b8 (127.6.61.2:42331)
I20260812 07:07:11.748116 22755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.2@KRBTEST.COM'} at 127.6.61.2:56543
WARNING: no policy specified for kudu/127.6.61.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
I20260812 07:07:11.798359 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.3:0
--local_ip_for_outbound_sockets=127.6.61.3
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:43861
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:45733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:07:12.153731 23141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:07:12.154073 23141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:07:12.154240 23141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:07:12.161667 23141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:07:12.161782 23141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:07:12.161976 23141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.3
I20260812 07:07:12.172449 23141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:45733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:43861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.3
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:07:12.173848 23141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:07:12.175190 23141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 07:07:12.188639 22972 heartbeater.cc:503] Master 127.6.61.62:43861 was elected leader, sending a full tablet report...
W20260812 07:07:12.189154 23148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:12.189209 23149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:12.190114 23151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:07:12.191087 23141 server_base.cc:1034] running on GCE node
I20260812 07:07:12.191843 23141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:07:12.192888 23141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:07:12.194119 23141 hybrid_clock.cc:648] HybridClock initialized: now 1786518432194074 us; error 72 us; skew 500 ppm
I20260812 07:07:12.194468 23141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:07:12 dist-test-slave-69rg krb5kdc[22702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518432, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:07:12.197873 23141 init.cc:377] Logged in from keytab as kudu/127.6.61.3@KRBTEST.COM (short username kudu)
I20260812 07:07:12.199491 23141 webserver.cc:460] Webserver started at http://127.6.61.3:41439/ using document root <none> and password file <none>
I20260812 07:07:12.200258 23141 fs_manager.cc:359] Metadata directory not provided
I20260812 07:07:12.200412 23141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:07:12.200852 23141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:07:12.203974 23141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/data/instance:
uuid: "4dacdbf4f80b4fe1bc0f0ce0af724579"
format_stamp: "Formatted at 2026-08-12 07:07:12 on dist-test-slave-69rg"
server_key: "6c8610a2d2256f713d77ec73c927d79c"
server_key_iv: "b3b74dd0b297ecd88ba2c248e43d9580"
server_key_version: "encryptionkey@0"
I20260812 07:07:12.204783 23141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance:
uuid: "4dacdbf4f80b4fe1bc0f0ce0af724579"
format_stamp: "Formatted at 2026-08-12 07:07:12 on dist-test-slave-69rg"
server_key: "6c8610a2d2256f713d77ec73c927d79c"
server_key_iv: "b3b74dd0b297ecd88ba2c248e43d9580"
server_key_version: "encryptionkey@0"
I20260812 07:07:12.210237 23141 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260812 07:07:12.214017 23158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:12.215363 23141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:07:12.215547 23141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/wal
uuid: "4dacdbf4f80b4fe1bc0f0ce0af724579"
format_stamp: "Formatted at 2026-08-12 07:07:12 on dist-test-slave-69rg"
server_key: "6c8610a2d2256f713d77ec73c927d79c"
server_key_iv: "b3b74dd0b297ecd88ba2c248e43d9580"
server_key_version: "encryptionkey@0"
I20260812 07:07:12.215757 23141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:07:12.235581 23141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:07:12.240197 23141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:07:12.240506 23141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:07:12.241598 23141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:07:12.243095 23141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:07:12.243211 23141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:12.243362 23141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:07:12.243434 23141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:12.279150 23141 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.3:38069
I20260812 07:07:12.279201 23285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.3:38069 every 8 connection(s)
I20260812 07:07:12.280443 23141 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
I20260812 07:07:12.280917 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 23141
I20260812 07:07:12.281134 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance
I20260812 07:07:12.281847 6388 external_mini_cluster.cc:1428] Setting key 46ac3a88f80f455b175dc659e30dfdb6
Aug 12 07:07:12 dist-test-slave-69rg krb5kdc[22702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518432, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:07:12.301731 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:12.283666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:49003 (local address 127.6.61.62:43861)
0812 07:07:12.284095 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:12.284100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:12.285513 (+ 1413us) server_negotiation.cc:407] Connection header received
0812 07:07:12.287425 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:12.287429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:12.287494 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:12.287589 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:12.289178 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:12.290048 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:12.291523 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:12.291912 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:12.292004 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:12.295331 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:12.295336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:12.295339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:12.295371 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:12.298809 (+ 3438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:12.299481 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:12.299484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:12.299487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:12.299612 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:12.300090 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:12.300093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:12.300095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:12.300411 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:12.300525 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:12.301283 (+ 758us) server_negotiation.cc:299] Negotiation successful
0812 07:07:12.301460 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":556,"threads_started":1}
I20260812 07:07:12.302659 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:12.284434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.3:49003)
0812 07:07:12.285339 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:12.285394 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:12.287173 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:12.287838 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:12.287873 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:12.288419 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:07:12.289015 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:12.289044 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:12.290179 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:12.290183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:12.291399 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:12.291412 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:12.292105 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:12.292110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:12.292316 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:12.292936 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:07:12.292963 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:07:12.295133 (+ 2170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:07:12.298928 (+ 3795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:12.298935 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:12.298946 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:12.299379 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:12.299723 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:12.299726 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:12.299728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:12.299900 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:12.300521 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:07:12.300527 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:07:12.300838 (+ 311us) client_negotiation.cc:769] Sending connection context
0812 07:07:12.301113 (+ 275us) client_negotiation.cc:240] Negotiation successful
0812 07:07:12.301528 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":394,"threads_started":1}
I20260812 07:07:12.304030 23286 heartbeater.cc:347] Connected to a master server at 127.6.61.62:43861
I20260812 07:07:12.304349 23286 heartbeater.cc:464] Registering TS with master...
I20260812 07:07:12.305085 23286 heartbeater.cc:511] Master 127.6.61.62:43861 requested a full tablet report, sending...
I20260812 07:07:12.306581 22755 ts_manager.cc:194] Registered new tserver with Master: 4dacdbf4f80b4fe1bc0f0ce0af724579 (127.6.61.3:38069)
I20260812 07:07:12.307402 22755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.3@KRBTEST.COM'} at 127.6.61.3:49003
I20260812 07:07:12.311045 6388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 07:07:12.330015 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:12.316178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49912 (local address 127.6.61.62:43861)
0812 07:07:12.316484 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:12.316493 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:12.316832 (+ 339us) server_negotiation.cc:407] Connection header received
0812 07:07:12.317074 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:12.317080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:12.317161 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:12.317281 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:12.318905 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:12.319724 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:12.321229 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:12.321646 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:12.321791 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:12.324418 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:12.324423 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:12.324427 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:12.324462 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:12.327135 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:12.327877 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:12.327884 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:12.327888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:12.328024 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:12.328428 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:12.328433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:12.328436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:12.328798 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:12.329073 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:12.329419 (+ 346us) server_negotiation.cc:299] Negotiation successful
0812 07:07:12.329667 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260812 07:07:12.334406 22755 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:07:12.341470 22755 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:07:12.376976 23309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:12.357215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55032 (local address 127.6.61.1:36915)
0812 07:07:12.363371 (+ 6156us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:12.363386 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:12.367577 (+ 4191us) server_negotiation.cc:407] Connection header received
0812 07:07:12.367731 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:12.367743 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:12.368106 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:12.368329 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:07:12.369561 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:12.371321 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:12.374228 (+ 2907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:12.375660 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:12.375872 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:12.376124 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:12.376415 (+ 291us) server_negotiation.cc:299] Negotiation successful
0812 07:07:12.376591 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5885,"thread_start_us":5691,"threads_started":1}
I20260812 07:07:12.377434 23306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:12.356434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:36915 (local address 127.0.0.1:55032)
0812 07:07:12.367233 (+ 10799us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:12.367254 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:12.367393 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:12.368665 (+ 1272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:12.368677 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:12.368717 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:07:12.369256 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:12.369273 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:12.371482 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:12.371488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:12.374045 (+ 2557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:12.374061 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:12.375915 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:12.375922 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:12.376111 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:12.376174 (+ 63us) client_negotiation.cc:769] Sending connection context
0812 07:07:12.376703 (+ 529us) client_negotiation.cc:240] Negotiation successful
0812 07:07:12.376867 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10631,"mutex_wait_us":56,"thread_start_us":7642,"threads_started":1}
I20260812 07:07:12.377948 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:12.354238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36792 (local address 127.6.61.3:38069)
0812 07:07:12.368760 (+ 14522us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:12.368773 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:12.368802 (+ 29us) server_negotiation.cc:407] Connection header received
0812 07:07:12.368878 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:12.368884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:12.369034 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:12.369241 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:07:12.370035 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:12.371995 (+ 1960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:12.374354 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:12.375837 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:12.375970 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:12.376190 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:12.377360 (+ 1170us) server_negotiation.cc:299] Negotiation successful
0812 07:07:12.377569 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14363,"thread_start_us":14182,"threads_started":1}
I20260812 07:07:12.378495 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:12.353992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:38069 (local address 127.0.0.1:36792)
0812 07:07:12.364603 (+ 10611us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:12.364668 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:12.364923 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:12.369235 (+ 4312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:12.369240 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:12.369274 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:07:12.369729 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:12.369742 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:12.372154 (+ 2412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:12.372160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:12.374085 (+ 1925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:12.374099 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:12.376302 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:12.376311 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:12.376519 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:12.376550 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 07:07:12.376643 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 07:07:12.376865 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10249,"mutex_wait_us":68,"thread_start_us":13123,"threads_started":1}
I20260812 07:07:12.378518 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:12.354407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:42331 (local address 127.0.0.1:42204)
0812 07:07:12.366471 (+ 12064us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:12.366494 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:12.366643 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:12.370439 (+ 3796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:12.370443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:12.370468 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:07:12.370893 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:12.370910 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:12.372615 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:12.372626 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:12.375214 (+ 2588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:12.375230 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:12.377466 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:12.377475 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:12.377645 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:12.377676 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 07:07:12.377808 (+ 132us) client_negotiation.cc:240] Negotiation successful
0812 07:07:12.377985 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11895,"mutex_wait_us":57,"thread_start_us":12494,"threads_started":1}
I20260812 07:07:12.379504 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:12.354579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42204 (local address 127.6.61.2:42331)
0812 07:07:12.369978 (+ 15399us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:12.369989 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:12.370024 (+ 35us) server_negotiation.cc:407] Connection header received
0812 07:07:12.370098 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:12.370103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:12.370258 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:12.370444 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:07:12.371094 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:12.372443 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:12.375412 (+ 2969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:12.376959 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:12.377098 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:12.377289 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:12.378400 (+ 1111us) server_negotiation.cc:299] Negotiation successful
0812 07:07:12.378578 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15193,"thread_start_us":15091,"threads_started":1}
I20260812 07:07:12.382073 22900 tablet_service.cc:1467] Processing CreateTablet for tablet 7197cbae10b94c66bb69767f266ef827 (DEFAULT_TABLE table=test-table [id=dc5d5ce3845342d48697dd30de2d0175]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:07:12.382108 23215 tablet_service.cc:1467] Processing CreateTablet for tablet 7197cbae10b94c66bb69767f266ef827 (DEFAULT_TABLE table=test-table [id=dc5d5ce3845342d48697dd30de2d0175]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:07:12.383008 23053 tablet_service.cc:1467] Processing CreateTablet for tablet 7197cbae10b94c66bb69767f266ef827 (DEFAULT_TABLE table=test-table [id=dc5d5ce3845342d48697dd30de2d0175]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:07:12.383708 23215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7197cbae10b94c66bb69767f266ef827. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:07:12.383772 22900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7197cbae10b94c66bb69767f266ef827. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:07:12.384392 23053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7197cbae10b94c66bb69767f266ef827. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:07:12.397137 23312 tablet_bootstrap.cc:492] T 7197cbae10b94c66bb69767f266ef827 P 4dacdbf4f80b4fe1bc0f0ce0af724579: Bootstrap starting.
I20260812 07:07:12.402207 23312 tablet_bootstrap.cc:654] T 7197cbae10b94c66bb69767f266ef827 P 4dacdbf4f80b4fe1bc0f0ce0af724579: Neither blocks nor log segments found. Creating new log.
I20260812 07:07:12.403337 23312 log.cc:824] T 7197cbae10b94c66bb69767f266ef827 P 4dacdbf4f80b4fe1bc0f0ce0af724579: Log is configured to *not* fsync() on all Append() calls
I20260812 07:07:12.406225 23314 tablet_bootstrap.cc:492] T 7197cbae10b94c66bb69767f266ef827 P c01e069207904975a3e673740a311e08: Bootstrap starting.
I20260812 07:07:12.406319 23313 tablet_bootstrap.cc:492] T 7197cbae10b94c66bb69767f266ef827 P 74c1eb79d71c44a5b1de64128768e0b8: Bootstrap starting.
I20260812 07:07:12.407050 23312 tablet_bootstrap.cc:492] T 7197cbae10b94c66bb69767f266ef827 P 4dacdbf4f80b4fe1bc0f0ce0af724579: No bootstrap required, opened a new log
I20260812 07:07:12.407366 23312 ts_tablet_manager.cc:1397] T 7197cbae10b94c66bb69767f266ef827 P 4dacdbf4f80b4fe1bc0f0ce0af724579: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260812 07:07:12.410632 23313 tablet_bootstrap.cc:654] T 7197cbae10b94c66bb69767f266ef827 P 74c1eb79d71c44a5b1de64128768e0b8: Neither blocks nor log segments found. Creating new log.
I20260812 07:07:12.411212 23314 tablet_bootstrap.cc:654] T 7197cbae10b94c66bb69767f266ef827 P c01e069207904975a3e673740a311e08: Neither blocks nor log segments found. Creating new log.
I20260812 07:07:12.412103 23313 log.cc:824] T 7197cbae10b94c66bb69767f266ef827 P 74c1eb79d71c44a5b1de64128768e0b8: Log is configured to *not* fsync() on all Append() calls
I20260812 07:07:12.412580 23314 log.cc:824] T 7197cbae10b94c66bb69767f266ef827 P c01e069207904975a3e673740a311e08: Log is configured to *not* fsync() on all Append() calls
I20260812 07:07:12.415666 23313 tablet_bootstrap.cc:492] T 7197cbae10b94c66bb69767f266ef827 P 74c1eb79d71c44a5b1de64128768e0b8: No bootstrap required, opened a new log
I20260812 07:07:12.415906 23314 tablet_bootstrap.cc:492] T 7197cbae10b94c66bb69767f266ef827 P c01e069207904975a3e673740a311e08: No bootstrap required, opened a new log
I20260812 07:07:12.416033 23313 ts_tablet_manager.cc:1397] T 7197cbae10b94c66bb69767f266ef827 P 74c1eb79d71c44a5b1de64128768e0b8: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260812 07:07:12.416199 23314 ts_tablet_manager.cc:1397] T 7197cbae10b94c66bb69767f266ef827 P c01e069207904975a3e673740a311e08: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260812 07:07:12.420960 23312 raft_consensus.cc:348] T 7197cbae10b94c66bb69767f266ef827 P 4dacdbf4f80b4fe1bc0f0ce0af724579 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dacdbf4f80b4fe1bc0f0ce0af724579" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38069 } } peers { permanent_uuid: "74c1eb79d71c44a5b1de64128768e0b8" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 42331 } } peers { permanent_uuid: "c01e069207904975a3e673740a311e08" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 36915 } }
I20260812 07:07:12.421424 23312 raft_consensus.cc:374] T 7197cbae10b94c66bb69767f266ef827 P 4dacdbf4f80b4fe1bc0f0ce0af724579 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:07:12.421571 23312 raft_consensus.cc:729] T 7197cbae10b94c66bb69767f266ef827 P 4dacdbf4f80b4fe1bc0f0ce0af724579 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dacdbf4f80b4fe1bc0f0ce0af724579, State: Initialized, Role: FOLLOWER
I20260812 07:07:12.422365 23312 consensus_queue.cc:260] T 7197cbae10b94c66bb69767f266ef827 P 4dacdbf4f80b4fe1bc0f0ce0af724579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dacdbf4f80b4fe1bc0f0ce0af724579" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38069 } } peers { permanent_uuid: "74c1eb79d71c44a5b1de64128768e0b8" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 42331 } } peers { permanent_uuid: "c01e069207904975a3e673740a311e08" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 36915 } }
I20260812 07:07:12.423642 23286 heartbeater.cc:503] Master 127.6.61.62:43861 was elected leader, sending a full tablet report...
I20260812 07:07:12.424391 23312 ts_tablet_manager.cc:1428] T 7197cbae10b94c66bb69767f266ef827 P 4dacdbf4f80b4fe1bc0f0ce0af724579: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
I20260812 07:07:12.425554 23313 raft_consensus.cc:348] T 7197cbae10b94c66bb69767f266ef827 P 74c1eb79d71c44a5b1de64128768e0b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dacdbf4f80b4fe1bc0f0ce0af724579" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38069 } } peers { permanent_uuid: "74c1eb79d71c44a5b1de64128768e0b8" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 42331 } } peers { permanent_uuid: "c01e069207904975a3e673740a311e08" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 36915 } }
I20260812 07:07:12.426213 23313 raft_consensus.cc:374] T 7197cbae10b94c66bb69767f266ef827 P 74c1eb79d71c44a5b1de64128768e0b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:07:12.426369 23313 raft_consensus.cc:729] T 7197cbae10b94c66bb69767f266ef827 P 74c1eb79d71c44a5b1de64128768e0b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c1eb79d71c44a5b1de64128768e0b8, State: Initialized, Role: FOLLOWER
I20260812 07:07:12.427230 23313 consensus_queue.cc:260] T 7197cbae10b94c66bb69767f266ef827 P 74c1eb79d71c44a5b1de64128768e0b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dacdbf4f80b4fe1bc0f0ce0af724579" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38069 } } peers { permanent_uuid: "74c1eb79d71c44a5b1de64128768e0b8" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 42331 } } peers { permanent_uuid: "c01e069207904975a3e673740a311e08" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 36915 } }
I20260812 07:07:12.427661 23314 raft_consensus.cc:348] T 7197cbae10b94c66bb69767f266ef827 P c01e069207904975a3e673740a311e08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dacdbf4f80b4fe1bc0f0ce0af724579" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38069 } } peers { permanent_uuid: "74c1eb79d71c44a5b1de64128768e0b8" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 42331 } } peers { permanent_uuid: "c01e069207904975a3e673740a311e08" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 36915 } }
I20260812 07:07:12.428186 23314 raft_consensus.cc:374] T 7197cbae10b94c66bb69767f266ef827 P c01e069207904975a3e673740a311e08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:07:12.428349 23314 raft_consensus.cc:729] T 7197cbae10b94c66bb69767f266ef827 P c01e069207904975a3e673740a311e08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c01e069207904975a3e673740a311e08, State: Initialized, Role: FOLLOWER
I20260812 07:07:12.428867 23133 heartbeater.cc:503] Master 127.6.61.62:43861 was elected leader, sending a full tablet report...
I20260812 07:07:12.429297 23313 ts_tablet_manager.cc:1428] T 7197cbae10b94c66bb69767f266ef827 P 74c1eb79d71c44a5b1de64128768e0b8: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260812 07:07:12.429123 23314 consensus_queue.cc:260] T 7197cbae10b94c66bb69767f266ef827 P c01e069207904975a3e673740a311e08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dacdbf4f80b4fe1bc0f0ce0af724579" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38069 } } peers { permanent_uuid: "74c1eb79d71c44a5b1de64128768e0b8" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 42331 } } peers { permanent_uuid: "c01e069207904975a3e673740a311e08" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 36915 } }
I20260812 07:07:12.430951 23314 ts_tablet_manager.cc:1428] T 7197cbae10b94c66bb69767f266ef827 P c01e069207904975a3e673740a311e08: Time spent starting tablet: real 0.015s user 0.000s sys 0.015s
W20260812 07:07:12.471532 23135 tablet.cc:2369] T 7197cbae10b94c66bb69767f266ef827 P 74c1eb79d71c44a5b1de64128768e0b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 07:07:12.532693 23287 tablet.cc:2369] T 7197cbae10b94c66bb69767f266ef827 P 4dacdbf4f80b4fe1bc0f0ce0af724579: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:07:12.570869 23321 raft_consensus.cc:482] T 7197cbae10b94c66bb69767f266ef827 P c01e069207904975a3e673740a311e08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:07:12.571179 23321 raft_consensus.cc:504] T 7197cbae10b94c66bb69767f266ef827 P c01e069207904975a3e673740a311e08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dacdbf4f80b4fe1bc0f0ce0af724579" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38069 } } peers { permanent_uuid: "74c1eb79d71c44a5b1de64128768e0b8" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 42331 } } peers { permanent_uuid: "c01e069207904975a3e673740a311e08" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 36915 } }
I20260812 07:07:12.572912 23321 leader_election.cc:288] T 7197cbae10b94c66bb69767f266ef827 P c01e069207904975a3e673740a311e08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4dacdbf4f80b4fe1bc0f0ce0af724579 (127.6.61.3:38069), 74c1eb79d71c44a5b1de64128768e0b8 (127.6.61.2:42331)
I20260812 07:07:12.582985 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:12.573559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:47691 (local address 127.6.61.2:42331)
0812 07:07:12.573759 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:12.573794 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:12.574669 (+ 875us) server_negotiation.cc:407] Connection header received
0812 07:07:12.574875 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:12.574879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:12.574961 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:12.575067 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:07:12.575967 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:12.577138 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:12.580349 (+ 3211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:12.581631 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:12.581786 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:12.581918 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:12.582530 (+ 612us) server_negotiation.cc:299] Negotiation successful
0812 07:07:12.582683 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 07:07:12.582985 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:12.573549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:44851 (local address 127.6.61.3:38069)
0812 07:07:12.573759 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:12.573766 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:12.574715 (+ 949us) server_negotiation.cc:407] Connection header received
0812 07:07:12.574873 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:12.574878 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:12.574961 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:12.575067 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:07:12.575967 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:12.577145 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:12.580350 (+ 3205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:12.581650 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:12.581786 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:12.581919 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:12.582530 (+ 611us) server_negotiation.cc:299] Negotiation successful
0812 07:07:12.582702 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 07:07:12.583276 23322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:12.573402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:38069 (local address 127.6.61.1:44851)
0812 07:07:12.574590 (+ 1188us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:12.574608 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:12.574718 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:12.575100 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:12.575107 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:12.575156 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:07:12.575789 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:12.575798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:12.577289 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:12.577295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:12.580125 (+ 2830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:12.580144 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:12.581983 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:12.581989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:12.582243 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:12.582343 (+ 100us) client_negotiation.cc:769] Sending connection context
0812 07:07:12.582449 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 07:07:12.582694 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1075,"mutex_wait_us":274,"spinlock_wait_cycles":14464,"thread_start_us":629,"threads_started":1}
I20260812 07:07:12.583360 23323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:12.573329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:42331 (local address 127.6.61.1:47691)
0812 07:07:12.574516 (+ 1187us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:12.574538 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:12.574714 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:12.575100 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:12.575107 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:12.575154 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:07:12.575762 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:12.575777 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:12.577312 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:12.577319 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:12.580125 (+ 2806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:12.580145 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:12.581970 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:12.581978 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:12.582245 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:12.582343 (+ 98us) client_negotiation.cc:769] Sending connection context
0812 07:07:12.582449 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 07:07:12.582694 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1021,"mutex_wait_us":363,"spinlock_wait_cycles":19584,"thread_start_us":717,"threads_started":1}
I20260812 07:07:12.584955 23236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7197cbae10b94c66bb69767f266ef827" candidate_uuid: "c01e069207904975a3e673740a311e08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dacdbf4f80b4fe1bc0f0ce0af724579" is_pre_election: true
I20260812 07:07:12.584955 23076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7197cbae10b94c66bb69767f266ef827" candidate_uuid: "c01e069207904975a3e673740a311e08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c1eb79d71c44a5b1de64128768e0b8" is_pre_election: true
I20260812 07:07:12.585619 23076 raft_consensus.cc:2454] T 7197cbae10b94c66bb69767f266ef827 P 74c1eb79d71c44a5b1de64128768e0b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c01e069207904975a3e673740a311e08 in term 0.
I20260812 07:07:12.585634 23236 raft_consensus.cc:2454] T 7197cbae10b94c66bb69767f266ef827 P 4dacdbf4f80b4fe1bc0f0ce0af724579 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c01e069207904975a3e673740a311e08 in term 0.
I20260812 07:07:12.586453 22845 leader_election.cc:302] T 7197cbae10b94c66bb69767f266ef827 P c01e069207904975a3e673740a311e08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4dacdbf4f80b4fe1bc0f0ce0af724579, 74c1eb79d71c44a5b1de64128768e0b8, c01e069207904975a3e673740a311e08; no voters:
I20260812 07:07:12.586997 23321 raft_consensus.cc:2781] T 7197cbae10b94c66bb69767f266ef827 P c01e069207904975a3e673740a311e08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:07:12.587172 23321 raft_consensus.cc:482] T 7197cbae10b94c66bb69767f266ef827 P c01e069207904975a3e673740a311e08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:07:12.587302 23321 raft_consensus.cc:3037] T 7197cbae10b94c66bb69767f266ef827 P c01e069207904975a3e673740a311e08 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:07:12.589599 23321 raft_consensus.cc:504] T 7197cbae10b94c66bb69767f266ef827 P c01e069207904975a3e673740a311e08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dacdbf4f80b4fe1bc0f0ce0af724579" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38069 } } peers { permanent_uuid: "74c1eb79d71c44a5b1de64128768e0b8" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 42331 } } peers { permanent_uuid: "c01e069207904975a3e673740a311e08" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 36915 } }
I20260812 07:07:12.590273 23321 leader_election.cc:288] T 7197cbae10b94c66bb69767f266ef827 P c01e069207904975a3e673740a311e08 [CANDIDATE]: Term 1 election: Requested vote from peers 4dacdbf4f80b4fe1bc0f0ce0af724579 (127.6.61.3:38069), 74c1eb79d71c44a5b1de64128768e0b8 (127.6.61.2:42331)
I20260812 07:07:12.590870 23076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7197cbae10b94c66bb69767f266ef827" candidate_uuid: "c01e069207904975a3e673740a311e08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c1eb79d71c44a5b1de64128768e0b8"
I20260812 07:07:12.590870 23236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7197cbae10b94c66bb69767f266ef827" candidate_uuid: "c01e069207904975a3e673740a311e08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dacdbf4f80b4fe1bc0f0ce0af724579"
I20260812 07:07:12.591089 23236 raft_consensus.cc:3037] T 7197cbae10b94c66bb69767f266ef827 P 4dacdbf4f80b4fe1bc0f0ce0af724579 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:07:12.591091 23076 raft_consensus.cc:3037] T 7197cbae10b94c66bb69767f266ef827 P 74c1eb79d71c44a5b1de64128768e0b8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:07:12.594412 23076 raft_consensus.cc:2454] T 7197cbae10b94c66bb69767f266ef827 P 74c1eb79d71c44a5b1de64128768e0b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c01e069207904975a3e673740a311e08 in term 1.
I20260812 07:07:12.594412 23236 raft_consensus.cc:2454] T 7197cbae10b94c66bb69767f266ef827 P 4dacdbf4f80b4fe1bc0f0ce0af724579 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c01e069207904975a3e673740a311e08 in term 1.
I20260812 07:07:12.594893 22845 leader_election.cc:302] T 7197cbae10b94c66bb69767f266ef827 P c01e069207904975a3e673740a311e08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4dacdbf4f80b4fe1bc0f0ce0af724579, 74c1eb79d71c44a5b1de64128768e0b8, c01e069207904975a3e673740a311e08; no voters:
I20260812 07:07:12.595258 23321 raft_consensus.cc:2781] T 7197cbae10b94c66bb69767f266ef827 P c01e069207904975a3e673740a311e08 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:07:12.595753 23321 raft_consensus.cc:686] T 7197cbae10b94c66bb69767f266ef827 P c01e069207904975a3e673740a311e08 [term 1 LEADER]: Becoming Leader. State: Replica: c01e069207904975a3e673740a311e08, State: Running, Role: LEADER
I20260812 07:07:12.596273 23321 consensus_queue.cc:237] T 7197cbae10b94c66bb69767f266ef827 P c01e069207904975a3e673740a311e08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dacdbf4f80b4fe1bc0f0ce0af724579" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38069 } } peers { permanent_uuid: "74c1eb79d71c44a5b1de64128768e0b8" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 42331 } } peers { permanent_uuid: "c01e069207904975a3e673740a311e08" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 36915 } }
I20260812 07:07:12.601485 22753 catalog_manager.cc:5495] T 7197cbae10b94c66bb69767f266ef827 P c01e069207904975a3e673740a311e08 reported cstate change: term changed from 0 to 1, leader changed from <none> to c01e069207904975a3e673740a311e08 (127.6.61.1). New cstate: current_term: 1 leader_uuid: "c01e069207904975a3e673740a311e08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dacdbf4f80b4fe1bc0f0ce0af724579" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74c1eb79d71c44a5b1de64128768e0b8" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 42331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c01e069207904975a3e673740a311e08" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 36915 } health_report { overall_health: HEALTHY } } }
I20260812 07:07:12.621126 23309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:12.615190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55040 (local address 127.6.61.1:36915)
0812 07:07:12.615411 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:12.615418 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:12.615437 (+ 19us) server_negotiation.cc:407] Connection header received
0812 07:07:12.615677 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:12.615681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:12.615778 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:12.615934 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:07:12.616351 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:12.617108 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:12.618815 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:12.619245 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:12.619353 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:12.619845 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:07:12.620466 (+ 621us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:07:12.620590 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:12.620708 (+ 118us) server_negotiation.cc:299] Negotiation successful
0812 07:07:12.620780 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 07:07:12.629611 23076 raft_consensus.cc:1261] T 7197cbae10b94c66bb69767f266ef827 P 74c1eb79d71c44a5b1de64128768e0b8 [term 1 FOLLOWER]: Refusing update from remote peer c01e069207904975a3e673740a311e08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:07:12.629639 23236 raft_consensus.cc:1261] T 7197cbae10b94c66bb69767f266ef827 P 4dacdbf4f80b4fe1bc0f0ce0af724579 [term 1 FOLLOWER]: Refusing update from remote peer c01e069207904975a3e673740a311e08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:07:12.630724 23325 consensus_queue.cc:1035] T 7197cbae10b94c66bb69767f266ef827 P c01e069207904975a3e673740a311e08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dacdbf4f80b4fe1bc0f0ce0af724579" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:07:12.631110 23321 consensus_queue.cc:1035] T 7197cbae10b94c66bb69767f266ef827 P c01e069207904975a3e673740a311e08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74c1eb79d71c44a5b1de64128768e0b8" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 42331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:07:12.644374 23328 mvcc.cc:205] Tried to move back new op lower bound from 7317579500040704000 to 7317579499918344192. Current Snapshot: MvccSnapshot[applied={T|T < 7317579500040704000}]
I20260812 07:07:12.665966 22753 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:07:12.666308 22753 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:07:12.670331 22753 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04: Sending DeleteTablet for 3 replicas of tablet 7197cbae10b94c66bb69767f266ef827
I20260812 07:07:12.671531 23053 tablet_service.cc:1514] Processing DeleteTablet for tablet 7197cbae10b94c66bb69767f266ef827 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:07:12 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:42204
I20260812 07:07:12.671550 22900 tablet_service.cc:1514] Processing DeleteTablet for tablet 7197cbae10b94c66bb69767f266ef827 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:07:12 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:55032
I20260812 07:07:12.671550 23215 tablet_service.cc:1514] Processing DeleteTablet for tablet 7197cbae10b94c66bb69767f266ef827 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:07:12 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:36792
I20260812 07:07:12.672832 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 22826
I20260812 07:07:12.673133 23342 tablet_replica.cc:330] T 7197cbae10b94c66bb69767f266ef827 P 4dacdbf4f80b4fe1bc0f0ce0af724579: stopping tablet replica
I20260812 07:07:12.673754 23342 raft_consensus.cc:2229] T 7197cbae10b94c66bb69767f266ef827 P 4dacdbf4f80b4fe1bc0f0ce0af724579 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:07:12.675009 23342 raft_consensus.cc:2258] T 7197cbae10b94c66bb69767f266ef827 P 4dacdbf4f80b4fe1bc0f0ce0af724579 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:07:12.676396 23341 tablet_replica.cc:330] T 7197cbae10b94c66bb69767f266ef827 P c01e069207904975a3e673740a311e08: stopping tablet replica
I20260812 07:07:12.676527 23340 tablet_replica.cc:330] T 7197cbae10b94c66bb69767f266ef827 P 74c1eb79d71c44a5b1de64128768e0b8: stopping tablet replica
I20260812 07:07:12.676819 23341 raft_consensus.cc:2229] T 7197cbae10b94c66bb69767f266ef827 P c01e069207904975a3e673740a311e08 [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:07:12.677002 23340 raft_consensus.cc:2229] T 7197cbae10b94c66bb69767f266ef827 P 74c1eb79d71c44a5b1de64128768e0b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:07:12.677529 23340 raft_consensus.cc:2258] T 7197cbae10b94c66bb69767f266ef827 P 74c1eb79d71c44a5b1de64128768e0b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:07:12.677529 23341 raft_consensus.cc:2258] T 7197cbae10b94c66bb69767f266ef827 P c01e069207904975a3e673740a311e08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:07:12.679697 23342 ts_tablet_manager.cc:1905] T 7197cbae10b94c66bb69767f266ef827 P 4dacdbf4f80b4fe1bc0f0ce0af724579: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:07:12.679705 23341 ts_tablet_manager.cc:1905] T 7197cbae10b94c66bb69767f266ef827 P c01e069207904975a3e673740a311e08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:07:12.681418 23340 ts_tablet_manager.cc:1905] T 7197cbae10b94c66bb69767f266ef827 P 74c1eb79d71c44a5b1de64128768e0b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:07:12.687801 23342 ts_tablet_manager.cc:1918] T 7197cbae10b94c66bb69767f266ef827 P 4dacdbf4f80b4fe1bc0f0ce0af724579: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:07:12.687990 23342 log.cc:1196] T 7197cbae10b94c66bb69767f266ef827 P 4dacdbf4f80b4fe1bc0f0ce0af724579: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/wal/wals/7197cbae10b94c66bb69767f266ef827
I20260812 07:07:12.688562 23342 ts_tablet_manager.cc:1939] T 7197cbae10b94c66bb69767f266ef827 P 4dacdbf4f80b4fe1bc0f0ce0af724579: Deleting consensus metadata
I20260812 07:07:12.690692 22736 catalog_manager.cc:4841] TS 4dacdbf4f80b4fe1bc0f0ce0af724579 (127.6.61.3:38069): tablet 7197cbae10b94c66bb69767f266ef827 (table test-table [id=dc5d5ce3845342d48697dd30de2d0175]) successfully deleted
I20260812 07:07:12.692526 23341 ts_tablet_manager.cc:1918] T 7197cbae10b94c66bb69767f266ef827 P c01e069207904975a3e673740a311e08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:07:12.692727 23341 log.cc:1196] T 7197cbae10b94c66bb69767f266ef827 P c01e069207904975a3e673740a311e08: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/wal/wals/7197cbae10b94c66bb69767f266ef827
I20260812 07:07:12.693327 23341 ts_tablet_manager.cc:1939] T 7197cbae10b94c66bb69767f266ef827 P c01e069207904975a3e673740a311e08: Deleting consensus metadata
Aug 12 07:07:12 dist-test-slave-69rg krb5kdc[22702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.1@KRBTEST.COM
I20260812 07:07:12.695235 22739 catalog_manager.cc:4841] TS c01e069207904975a3e673740a311e08 (127.6.61.1:36915): tablet 7197cbae10b94c66bb69767f266ef827 (table test-table [id=dc5d5ce3845342d48697dd30de2d0175]) successfully deleted
I20260812 07:07:12.696126 23340 ts_tablet_manager.cc:1918] T 7197cbae10b94c66bb69767f266ef827 P 74c1eb79d71c44a5b1de64128768e0b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:07:12.696300 23340 log.cc:1196] T 7197cbae10b94c66bb69767f266ef827 P 74c1eb79d71c44a5b1de64128768e0b8: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/wal/wals/7197cbae10b94c66bb69767f266ef827
I20260812 07:07:12.696866 23340 ts_tablet_manager.cc:1939] T 7197cbae10b94c66bb69767f266ef827 P 74c1eb79d71c44a5b1de64128768e0b8: Deleting consensus metadata
I20260812 07:07:12.698939 22737 catalog_manager.cc:4841] TS 74c1eb79d71c44a5b1de64128768e0b8 (127.6.61.2:42331): tablet 7197cbae10b94c66bb69767f266ef827 (table test-table [id=dc5d5ce3845342d48697dd30de2d0175]) successfully deleted
I20260812 07:07:12.703455 23309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:12.674154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55050 (local address 127.6.61.1:36915)
0812 07:07:12.674411 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:12.674420 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:12.682482 (+ 8062us) server_negotiation.cc:407] Connection header received
0812 07:07:12.682711 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:12.682716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:12.682837 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:12.682955 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:12.685101 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:12.686451 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:12.688685 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:12.689476 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:12.689576 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:12.695784 (+ 6208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:12.695802 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:12.695825 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:12.695889 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:12.699834 (+ 3945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:12.700844 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:12.700853 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:12.700856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:12.701020 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:12.701506 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:12.701511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:12.701513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:12.702135 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:12.702496 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:12.702847 (+ 351us) server_negotiation.cc:299] Negotiation successful
0812 07:07:12.703059 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 07:07:12.704377 22966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:07:12.804988 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 22826
I20260812 07:07:12.830701 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 22979
Aug 12 07:07:12 dist-test-slave-69rg krb5kdc[22702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.2@KRBTEST.COM
I20260812 07:07:12.846990 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:12.831628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42212 (local address 127.6.61.2:42331)
0812 07:07:12.831850 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:12.831856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:12.831876 (+ 20us) server_negotiation.cc:407] Connection header received
0812 07:07:12.832057 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:12.832062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:12.832208 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:12.832327 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:12.833735 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:12.834567 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:12.835885 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:12.836307 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:12.836449 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:12.841209 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:12.841227 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:12.841245 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:12.841290 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:12.844295 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:12.845026 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:12.845034 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:12.845037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:12.845189 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:12.845563 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:12.845566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:12.845568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:12.846061 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:12.846339 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:12.846544 (+ 205us) server_negotiation.cc:299] Negotiation successful
0812 07:07:12.846716 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 07:07:12.847875 23123 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:07:12.938684 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 22979
I20260812 07:07:12.960425 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 23141
Aug 12 07:07:12 dist-test-slave-69rg krb5kdc[22702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.3@KRBTEST.COM
I20260812 07:07:12.977567 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:12.961360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36796 (local address 127.6.61.3:38069)
0812 07:07:12.961547 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:12.961552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:12.961683 (+ 131us) server_negotiation.cc:407] Connection header received
0812 07:07:12.961863 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:12.961867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:12.961940 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:12.962028 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:12.963698 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:12.964456 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:12.966279 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:12.966687 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:12.966810 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:12.971929 (+ 5119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:12.971946 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:12.971964 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:12.972007 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:12.975016 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:12.975655 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:12.975662 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:12.975665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:12.975809 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:12.976133 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:12.976136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:12.976151 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:12.976496 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:12.976717 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:12.976983 (+ 266us) server_negotiation.cc:299] Negotiation successful
0812 07:07:12.977193 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 07:07:12.978368 23280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:07:13.071381 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 23141
I20260812 07:07:13.094586 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 22719
I20260812 07:07:13.095920 22787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:07:13.211366 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 22719
I20260812 07:07:13.237598 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:43861
--webserver_interface=127.6.61.62
--webserver_port=36017
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:45733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:43861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:07:13.599813 23350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:07:13.600250 23350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:07:13.600414 23350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:07:13.608760 23350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:07:13.608891 23350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:07:13.608981 23350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:07:13.609089 23350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:07:13.609181 23350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:07:13.619875 23350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:45733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:43861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:43861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=36017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:07:13.621367 23350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:07:13.622758 23350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:07:13.649138 23360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:13.649147 23358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:13.649286 23357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:07:13.649966 23350 server_base.cc:1034] running on GCE node
I20260812 07:07:13.650903 23350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:07:13.652629 23350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:07:13.653920 23350 hybrid_clock.cc:648] HybridClock initialized: now 1786518433653848 us; error 55 us; skew 500 ppm
I20260812 07:07:13.654346 23350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:07:13 dist-test-slave-69rg krb5kdc[22702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518433, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:07:13.658421 23350 init.cc:377] Logged in from keytab as kudu/127.6.61.62@KRBTEST.COM (short username kudu)
I20260812 07:07:13.660135 23350 webserver.cc:460] Webserver started at http://127.6.61.62:36017/ using document root <none> and password file <none>
I20260812 07:07:13.660905 23350 fs_manager.cc:359] Metadata directory not provided
I20260812 07:07:13.661034 23350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:07:13.667390 23350 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 07:07:13.670760 23368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:13.672422 23350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:07:13.672662 23350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "de11599be8e84615bc177368a6e7fb04"
format_stamp: "Formatted at 2026-08-12 07:07:10 on dist-test-slave-69rg"
server_key: "c35c551071f6d503fabb514bea3aaba4"
server_key_iv: "6d74cb59effa8d5cf2cf2841f2d1c9f5"
server_key_version: "encryptionkey@0"
I20260812 07:07:13.673903 23350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:07:13.699721 23350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:07:13.705056 23350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:07:13.705410 23350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:07:13.726117 23350 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:43861
I20260812 07:07:13.726138 23426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:43861 every 8 connection(s)
I20260812 07:07:13.727831 23350 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:07:13.728423 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 23350
I20260812 07:07:13.729503 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.1:36915
--local_ip_for_outbound_sockets=127.6.61.1
--tserver_master_addrs=127.6.61.62:43861
--webserver_port=46173
--webserver_interface=127.6.61.1
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:45733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:07:13.735780 23428 sys_catalog.cc:263] Verifying existing consensus state
I20260812 07:07:13.739789 23428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04: Bootstrap starting.
I20260812 07:07:13.768208 23428 log.cc:824] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04: Log is configured to *not* fsync() on all Append() calls
I20260812 07:07:13.784557 23428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 07:07:13.785274 23428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04: Bootstrap complete.
I20260812 07:07:13.795193 23428 raft_consensus.cc:348] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de11599be8e84615bc177368a6e7fb04" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 43861 } }
I20260812 07:07:13.796119 23428 raft_consensus.cc:729] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: de11599be8e84615bc177368a6e7fb04, State: Initialized, Role: FOLLOWER
I20260812 07:07:13.796753 23428 consensus_queue.cc:260] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de11599be8e84615bc177368a6e7fb04" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 43861 } }
I20260812 07:07:13.797029 23428 raft_consensus.cc:388] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:07:13.797168 23428 raft_consensus.cc:482] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:07:13.797363 23428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [term 1 FOLLOWER]: Advancing to term 2
I20260812 07:07:13.801416 23428 raft_consensus.cc:504] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de11599be8e84615bc177368a6e7fb04" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 43861 } }
I20260812 07:07:13.802037 23428 leader_election.cc:302] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de11599be8e84615bc177368a6e7fb04; no voters:
I20260812 07:07:13.802747 23428 leader_election.cc:288] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 07:07:13.802881 23433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 07:07:13.804127 23433 raft_consensus.cc:686] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [term 2 LEADER]: Becoming Leader. State: Replica: de11599be8e84615bc177368a6e7fb04, State: Running, Role: LEADER
I20260812 07:07:13.804697 23433 consensus_queue.cc:237] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de11599be8e84615bc177368a6e7fb04" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 43861 } }
I20260812 07:07:13.805276 23428 sys_catalog.cc:564] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:07:13.807509 23435 sys_catalog.cc:455] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de11599be8e84615bc177368a6e7fb04. Latest consensus state: current_term: 2 leader_uuid: "de11599be8e84615bc177368a6e7fb04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de11599be8e84615bc177368a6e7fb04" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 43861 } } }
I20260812 07:07:13.807966 23435 sys_catalog.cc:458] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [sys.catalog]: This master's current role is: LEADER
I20260812 07:07:13.808574 23434 sys_catalog.cc:455] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "de11599be8e84615bc177368a6e7fb04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de11599be8e84615bc177368a6e7fb04" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 43861 } } }
I20260812 07:07:13.808941 23434 sys_catalog.cc:458] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [sys.catalog]: This master's current role is: LEADER
I20260812 07:07:13.811242 23438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:07:13.819759 23438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:07:13.821224 23438 catalog_manager.cc:1259] Loaded cluster ID: 137db745a3ca4d959d8b8dc6c9f5b847
I20260812 07:07:13.821342 23438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:07:13.825160 23438 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260812 07:07:13.844347 23438 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260812 07:07:13.861888 23438 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:07:13.864137 23438 catalog_manager.cc:5879] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04: Loaded TSK: 0
I20260812 07:07:13.865762 23438 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 07:07:14.120527 23430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:07:14.120822 23430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:07:14.120990 23430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:07:14.128605 23430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:07:14.128718 23430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:07:14.128893 23430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.1
I20260812 07:07:14.138955 23430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:45733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.1:36915
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.61.1
--webserver_port=46173
--tserver_master_addrs=127.6.61.62:43861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.1
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:07:14.140342 23430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:07:14.141693 23430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:07:14.156857 23461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:14.156852 23460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:14.157656 23466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:07:14.158254 23430 server_base.cc:1034] running on GCE node
I20260812 07:07:14.158908 23430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:07:14.159904 23430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:07:14.161144 23430 hybrid_clock.cc:648] HybridClock initialized: now 1786518434161083 us; error 51 us; skew 500 ppm
I20260812 07:07:14.161482 23430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:07:14 dist-test-slave-69rg krb5kdc[22702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518434, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:07:14.164795 23430 init.cc:377] Logged in from keytab as kudu/127.6.61.1@KRBTEST.COM (short username kudu)
I20260812 07:07:14.166471 23430 webserver.cc:460] Webserver started at http://127.6.61.1:46173/ using document root <none> and password file <none>
I20260812 07:07:14.167209 23430 fs_manager.cc:359] Metadata directory not provided
I20260812 07:07:14.167380 23430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:07:14.173429 23430 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 07:07:14.176647 23473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:14.178249 23430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 07:07:14.178439 23430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/wal
uuid: "c01e069207904975a3e673740a311e08"
format_stamp: "Formatted at 2026-08-12 07:07:11 on dist-test-slave-69rg"
server_key: "aec759c06f64446dd7d3082a926807da"
server_key_iv: "58116310d115f420baf254d11a6a550b"
server_key_version: "encryptionkey@0"
I20260812 07:07:14.179560 23430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:07:14.194808 23430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:07:14.200312 23430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:07:14.200670 23430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:07:14.201771 23430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:07:14.203362 23430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:07:14.203513 23430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:14.203651 23430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:07:14.203747 23430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:14.240113 23430 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.1:36915
I20260812 07:07:14.240164 23603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.1:36915 every 8 connection(s)
I20260812 07:07:14.241926 23430 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
I20260812 07:07:14.247532 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 23430
I20260812 07:07:14.248369 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.2:42331
--local_ip_for_outbound_sockets=127.6.61.2
--tserver_master_addrs=127.6.61.62:43861
--webserver_port=39753
--webserver_interface=127.6.61.2
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:45733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 07:07:14 dist-test-slave-69rg krb5kdc[22702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518434, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:07:14.270074 23610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:14.245229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.1:38073)
0812 07:07:14.246124 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:14.246169 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:14.248621 (+ 2452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:14.250071 (+ 1450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:14.250084 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:14.250881 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:07:14.251669 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:14.251696 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:14.253350 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:14.253358 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:14.255595 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:14.255615 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:14.256653 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:14.256663 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:14.256974 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:14.257763 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:07:14.257798 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:07:14.260627 (+ 2829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:07:14.265524 (+ 4897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:14.265532 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:14.265544 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:14.266084 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:14.266620 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:14.266623 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:14.266625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:14.266810 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:14.267799 (+ 989us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:07:14.267805 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:07:14.268108 (+ 303us) client_negotiation.cc:769] Sending connection context
0812 07:07:14.268349 (+ 241us) client_negotiation.cc:240] Negotiation successful
0812 07:07:14.268759 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":375,"threads_started":1}
I20260812 07:07:14.270473 23608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:14.245116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:38073 (local address 127.6.61.62:43861)
0812 07:07:14.245884 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:14.245901 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:14.246308 (+ 407us) server_negotiation.cc:407] Connection header received
0812 07:07:14.249188 (+ 2880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:14.249202 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:14.249528 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:14.249904 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:14.251911 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:14.253166 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:14.255749 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:14.256404 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:14.256537 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:14.260963 (+ 4426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:14.260974 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:14.260997 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:14.261056 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:14.265345 (+ 4289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:14.266252 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:14.266264 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:14.266275 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:14.266501 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:14.266934 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:14.266941 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:14.266947 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:14.267651 (+ 704us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:14.267867 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:14.268660 (+ 793us) server_negotiation.cc:299] Negotiation successful
0812 07:07:14.269135 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":299,"threads_started":1}
I20260812 07:07:14.273274 23606 heartbeater.cc:347] Connected to a master server at 127.6.61.62:43861
I20260812 07:07:14.273623 23606 heartbeater.cc:464] Registering TS with master...
I20260812 07:07:14.274605 23606 heartbeater.cc:511] Master 127.6.61.62:43861 requested a full tablet report, sending...
I20260812 07:07:14.277169 23386 ts_manager.cc:194] Registered new tserver with Master: c01e069207904975a3e673740a311e08 (127.6.61.1:36915)
I20260812 07:07:14.278872 23386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.1@KRBTEST.COM'} at 127.6.61.1:38073
W20260812 07:07:14.596704 23611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:07:14.596987 23611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:07:14.597129 23611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:07:14.604553 23611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:07:14.604688 23611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:07:14.604877 23611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.2
I20260812 07:07:14.615118 23611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:45733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.2:42331
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.61.2
--webserver_port=39753
--tserver_master_addrs=127.6.61.62:43861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.2
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:07:14.616473 23611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:07:14.617871 23611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:07:14.631641 23620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:14.631703 23618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:14.631604 23617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:07:14.632829 23611 server_base.cc:1034] running on GCE node
I20260812 07:07:14.633488 23611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:07:14.634594 23611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:07:14.635841 23611 hybrid_clock.cc:648] HybridClock initialized: now 1786518434635773 us; error 61 us; skew 500 ppm
I20260812 07:07:14.636185 23611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:07:14 dist-test-slave-69rg krb5kdc[22702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518434, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:07:14.639669 23611 init.cc:377] Logged in from keytab as kudu/127.6.61.2@KRBTEST.COM (short username kudu)
I20260812 07:07:14.641220 23611 webserver.cc:460] Webserver started at http://127.6.61.2:39753/ using document root <none> and password file <none>
I20260812 07:07:14.642062 23611 fs_manager.cc:359] Metadata directory not provided
I20260812 07:07:14.642227 23611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:07:14.647966 23611 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260812 07:07:14.650892 23627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:14.652281 23611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 07:07:14.652449 23611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/wal
uuid: "74c1eb79d71c44a5b1de64128768e0b8"
format_stamp: "Formatted at 2026-08-12 07:07:11 on dist-test-slave-69rg"
server_key: "00bcc27a935bc7bc906722eeeb848a12"
server_key_iv: "b5df1704f6660f6017915cf24c0b7c27"
server_key_version: "encryptionkey@0"
I20260812 07:07:14.653434 23611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:07:14.680132 23611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:07:14.684731 23611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:07:14.685029 23611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:07:14.686131 23611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:07:14.687613 23611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:07:14.687711 23611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:14.687888 23611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:07:14.687952 23611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:14.725538 23611 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.2:42331
I20260812 07:07:14.725625 23761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.2:42331 every 8 connection(s)
I20260812 07:07:14.727099 23611 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
I20260812 07:07:14.734206 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 23611
I20260812 07:07:14.734859 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.3:38069
--local_ip_for_outbound_sockets=127.6.61.3
--tserver_master_addrs=127.6.61.62:43861
--webserver_port=41439
--webserver_interface=127.6.61.3
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:45733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 07:07:14 dist-test-slave-69rg krb5kdc[22702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518434, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:07:14.752367 23608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:14.729793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:56461 (local address 127.6.61.62:43861)
0812 07:07:14.730021 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:14.730028 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:14.731361 (+ 1333us) server_negotiation.cc:407] Connection header received
0812 07:07:14.733992 (+ 2631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:14.733999 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:14.734118 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:14.734247 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:14.736248 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:14.737432 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:14.739862 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:14.740707 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:14.740872 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:14.745652 (+ 4780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:14.745657 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:14.745661 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:14.745707 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:14.749195 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:14.750020 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:14.750024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:14.750026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:14.750192 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:14.750642 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:14.750646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:14.750650 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:14.750918 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:14.751078 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:14.751780 (+ 702us) server_negotiation.cc:299] Negotiation successful
0812 07:07:14.752018 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260812 07:07:14.753367 23765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:14.730335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.2:56461)
0812 07:07:14.731168 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:14.731216 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:14.733650 (+ 2434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:14.734527 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:14.734539 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:14.735293 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:07:14.736027 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:14.736049 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:14.737631 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:14.737637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:14.739684 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:14.739700 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:14.740978 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:14.740986 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:14.741297 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:14.742182 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:07:14.742213 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:07:14.745414 (+ 3201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:07:14.749351 (+ 3937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:14.749358 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:14.749371 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:14.749801 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:14.750313 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:14.750318 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:14.750322 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:14.750509 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:14.751051 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:07:14.751059 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:07:14.751397 (+ 338us) client_negotiation.cc:769] Sending connection context
0812 07:07:14.751676 (+ 279us) client_negotiation.cc:240] Negotiation successful
0812 07:07:14.752216 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":325,"threads_started":1}
I20260812 07:07:14.754925 23762 heartbeater.cc:347] Connected to a master server at 127.6.61.62:43861
I20260812 07:07:14.755307 23762 heartbeater.cc:464] Registering TS with master...
I20260812 07:07:14.756031 23762 heartbeater.cc:511] Master 127.6.61.62:43861 requested a full tablet report, sending...
I20260812 07:07:14.757583 23386 ts_manager.cc:194] Registered new tserver with Master: 74c1eb79d71c44a5b1de64128768e0b8 (127.6.61.2:42331)
I20260812 07:07:14.758929 23386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.2@KRBTEST.COM'} at 127.6.61.2:56461
W20260812 07:07:15.090983 23766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:07:15.091277 23766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:07:15.091377 23766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:07:15.098508 23766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:07:15.098613 23766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:07:15.098724 23766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.3
I20260812 07:07:15.108374 23766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:45733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.3:38069
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.61.3
--webserver_port=41439
--tserver_master_addrs=127.6.61.62:43861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.3
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:07:15.109661 23766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:07:15.110998 23766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:07:15.124851 23774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:15.124826 23775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:15.125036 23777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:07:15.125756 23766 server_base.cc:1034] running on GCE node
I20260812 07:07:15.126389 23766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:07:15.127338 23766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:07:15.128578 23766 hybrid_clock.cc:648] HybridClock initialized: now 1786518435128521 us; error 49 us; skew 500 ppm
I20260812 07:07:15.128914 23766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:07:15 dist-test-slave-69rg krb5kdc[22702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518435, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:07:15.132283 23766 init.cc:377] Logged in from keytab as kudu/127.6.61.3@KRBTEST.COM (short username kudu)
I20260812 07:07:15.133741 23766 webserver.cc:460] Webserver started at http://127.6.61.3:41439/ using document root <none> and password file <none>
I20260812 07:07:15.134450 23766 fs_manager.cc:359] Metadata directory not provided
I20260812 07:07:15.134567 23766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:07:15.139907 23766 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 07:07:15.142673 23784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:15.144016 23766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:07:15.144176 23766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/wal
uuid: "4dacdbf4f80b4fe1bc0f0ce0af724579"
format_stamp: "Formatted at 2026-08-12 07:07:12 on dist-test-slave-69rg"
server_key: "6c8610a2d2256f713d77ec73c927d79c"
server_key_iv: "b3b74dd0b297ecd88ba2c248e43d9580"
server_key_version: "encryptionkey@0"
I20260812 07:07:15.145100 23766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:07:15.160151 23766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:07:15.164630 23766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:07:15.164927 23766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:07:15.166070 23766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:07:15.167433 23766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:07:15.167572 23766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:15.167711 23766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:07:15.167821 23766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:15.203734 23766 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.3:38069
I20260812 07:07:15.203837 23915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.3:38069 every 8 connection(s)
I20260812 07:07:15.205338 23766 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
I20260812 07:07:15.209087 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 23766
Aug 12 07:07:15 dist-test-slave-69rg krb5kdc[22702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518435, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:07:15.233493 23608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:15.208277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:42785 (local address 127.6.61.62:43861)
0812 07:07:15.208468 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:15.208473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:15.210772 (+ 2299us) server_negotiation.cc:407] Connection header received
0812 07:07:15.214603 (+ 3831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:15.214614 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:15.214737 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:15.214883 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:15.217399 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.219012 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.220796 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.221385 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.221543 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:15.225609 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:15.225618 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:15.225625 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:15.225678 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:15.229599 (+ 3921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:15.230616 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:15.230624 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:15.230630 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:15.230804 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:15.231361 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:15.231365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:15.231367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:15.231704 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:15.231830 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:15.232813 (+ 983us) server_negotiation.cc:299] Negotiation successful
0812 07:07:15.233065 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":55}
I20260812 07:07:15.234591 23918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:15.208882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.3:42785)
0812 07:07:15.210563 (+ 1681us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:15.210622 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:15.214181 (+ 3559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:15.215195 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:15.215210 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:15.216173 (+ 963us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:07:15.217159 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:15.217189 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.219197 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.219203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:15.220625 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:15.220639 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.221661 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.221670 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:15.222032 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:15.222864 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:07:15.222890 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:07:15.225359 (+ 2469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:07:15.229764 (+ 4405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:15.229778 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:15.229800 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:15.230435 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:15.230942 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:15.230948 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:15.230954 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:15.231225 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:15.231847 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:07:15.231857 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:07:15.232310 (+ 453us) client_negotiation.cc:769] Sending connection context
0812 07:07:15.232624 (+ 314us) client_negotiation.cc:240] Negotiation successful
0812 07:07:15.233034 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1338,"thread_start_us":1092,"threads_started":1}
I20260812 07:07:15.235952 23919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:15.210437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49914 (local address 127.6.61.62:43861)
0812 07:07:15.218941 (+ 8504us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:15.218945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:15.218959 (+ 14us) server_negotiation.cc:407] Connection header received
0812 07:07:15.219002 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:15.219006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:15.219058 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:15.219180 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:15.220762 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.221634 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.223310 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.223759 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.223899 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:15.228632 (+ 4733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:15.228641 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:15.228646 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:15.228852 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:15.232391 (+ 3539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:15.233361 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:15.233367 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:15.233369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:15.233518 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:15.234129 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:15.234134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:15.234137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:15.234484 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:15.234758 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:15.235311 (+ 553us) server_negotiation.cc:299] Negotiation successful
0812 07:07:15.235553 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":8392,"thread_start_us":8250,"threads_started":1}
I20260812 07:07:15.236409 23916 heartbeater.cc:347] Connected to a master server at 127.6.61.62:43861
I20260812 07:07:15.236765 23916 heartbeater.cc:464] Registering TS with master...
I20260812 07:07:15.237615 23916 heartbeater.cc:511] Master 127.6.61.62:43861 requested a full tablet report, sending...
I20260812 07:07:15.239393 23386 ts_manager.cc:194] Registered new tserver with Master: 4dacdbf4f80b4fe1bc0f0ce0af724579 (127.6.61.3:38069)
I20260812 07:07:15.240252 23386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.3@KRBTEST.COM'} at 127.6.61.3:42785
I20260812 07:07:15.251684 6388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 07:07:15.259583 23919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:15.252887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49924 (local address 127.6.61.62:43861)
0812 07:07:15.253089 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:15.253094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:15.253525 (+ 431us) server_negotiation.cc:407] Connection header received
0812 07:07:15.253736 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:15.253741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:15.253962 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:15.254102 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:07:15.254594 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.255333 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.256891 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.257430 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.257652 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:15.258389 (+ 737us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:07:15.258988 (+ 599us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:07:15.259116 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:15.259278 (+ 162us) server_negotiation.cc:299] Negotiation successful
0812 07:07:15.259356 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 07:07:15.260715 23386 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:49924:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:07:15.268456 23386 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:07:15.282904 23606 heartbeater.cc:503] Master 127.6.61.62:43861 was elected leader, sending a full tablet report...
I20260812 07:07:15.304791 23927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:15.282408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:36915 (local address 127.0.0.1:55066)
0812 07:07:15.295860 (+ 13452us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:15.295885 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:15.296022 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:15.296899 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:15.296911 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:15.296966 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:07:15.297459 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:15.297475 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.299598 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.299604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:15.302111 (+ 2507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:15.302132 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.303783 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.303789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:15.303990 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:15.304044 (+ 54us) client_negotiation.cc:769] Sending connection context
0812 07:07:15.304186 (+ 142us) client_negotiation.cc:240] Negotiation successful
0812 07:07:15.304389 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13323,"thread_start_us":11097,"threads_started":1}
I20260812 07:07:15.304829 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:15.283176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55066 (local address 127.6.61.1:36915)
0812 07:07:15.292359 (+ 9183us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:15.292373 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:15.296165 (+ 3792us) server_negotiation.cc:407] Connection header received
0812 07:07:15.296281 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:15.296290 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:15.296615 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:15.296826 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:07:15.297741 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.299442 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.302328 (+ 2886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.303391 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.303472 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:15.303744 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:15.304269 (+ 525us) server_negotiation.cc:299] Negotiation successful
0812 07:07:15.304461 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8967,"thread_start_us":8784,"threads_started":1}
I20260812 07:07:15.306062 23931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:15.282931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36808 (local address 127.6.61.3:38069)
0812 07:07:15.296318 (+ 13387us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:15.296327 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:15.296922 (+ 595us) server_negotiation.cc:407] Connection header received
0812 07:07:15.297282 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:15.297290 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:15.297502 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:15.297684 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:07:15.298759 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.300121 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.302902 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.304390 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.304521 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:15.304736 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:15.305450 (+ 714us) server_negotiation.cc:299] Negotiation successful
0812 07:07:15.305637 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13199,"thread_start_us":13024,"threads_started":1}
I20260812 07:07:15.306252 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:15.281923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:42331 (local address 127.0.0.1:42214)
0812 07:07:15.294503 (+ 12580us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:15.294549 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:15.294744 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:15.298417 (+ 3673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:15.298424 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:15.298447 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:07:15.298858 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:15.298867 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.300763 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.300771 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:15.303449 (+ 2678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:15.303464 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.305306 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.305314 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:15.305523 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:15.305562 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 07:07:15.305649 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 07:07:15.305789 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11747,"thread_start_us":13662,"threads_started":1}
I20260812 07:07:15.305883 23930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:15.282488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:38069 (local address 127.0.0.1:36808)
0812 07:07:15.296816 (+ 14328us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:15.296835 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:15.297045 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:15.298047 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:15.298051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:15.298099 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:07:15.298581 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:15.298601 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.300260 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.300268 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:15.302714 (+ 2446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:15.302729 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.304921 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.304929 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:15.305163 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:15.305196 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 07:07:15.305290 (+ 94us) client_negotiation.cc:240] Negotiation successful
0812 07:07:15.305436 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14171,"mutex_wait_us":36,"thread_start_us":13998,"threads_started":1}
I20260812 07:07:15.306689 23928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:15.282039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42214 (local address 127.6.61.2:42331)
0812 07:07:15.297764 (+ 15725us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:15.297772 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:15.297802 (+ 30us) server_negotiation.cc:407] Connection header received
0812 07:07:15.297931 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:15.297939 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:15.298175 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:15.299017 (+ 842us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:07:15.299179 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.300600 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.303636 (+ 3036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.305042 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.305225 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:15.306145 (+ 920us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:15.306231 (+ 86us) server_negotiation.cc:299] Negotiation successful
0812 07:07:15.306406 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15560,"thread_start_us":15198,"threads_started":1}
I20260812 07:07:15.309690 23531 tablet_service.cc:1467] Processing CreateTablet for tablet 42abd31269694419a6624e582d5f333d (DEFAULT_TABLE table=test-table [id=372f4e65677247079f7a2da16a53abed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:07:15.310268 23835 tablet_service.cc:1467] Processing CreateTablet for tablet 42abd31269694419a6624e582d5f333d (DEFAULT_TABLE table=test-table [id=372f4e65677247079f7a2da16a53abed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:07:15.310268 23687 tablet_service.cc:1467] Processing CreateTablet for tablet 42abd31269694419a6624e582d5f333d (DEFAULT_TABLE table=test-table [id=372f4e65677247079f7a2da16a53abed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:07:15.311210 23531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42abd31269694419a6624e582d5f333d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:07:15.311606 23835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42abd31269694419a6624e582d5f333d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:07:15.311609 23687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42abd31269694419a6624e582d5f333d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:07:15.324829 23935 tablet_bootstrap.cc:492] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08: Bootstrap starting.
I20260812 07:07:15.327491 23937 tablet_bootstrap.cc:492] T 42abd31269694419a6624e582d5f333d P 74c1eb79d71c44a5b1de64128768e0b8: Bootstrap starting.
I20260812 07:07:15.330066 23935 tablet_bootstrap.cc:654] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08: Neither blocks nor log segments found. Creating new log.
I20260812 07:07:15.330725 23936 tablet_bootstrap.cc:492] T 42abd31269694419a6624e582d5f333d P 4dacdbf4f80b4fe1bc0f0ce0af724579: Bootstrap starting.
I20260812 07:07:15.331251 23937 tablet_bootstrap.cc:654] T 42abd31269694419a6624e582d5f333d P 74c1eb79d71c44a5b1de64128768e0b8: Neither blocks nor log segments found. Creating new log.
I20260812 07:07:15.331581 23935 log.cc:824] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08: Log is configured to *not* fsync() on all Append() calls
I20260812 07:07:15.332588 23937 log.cc:824] T 42abd31269694419a6624e582d5f333d P 74c1eb79d71c44a5b1de64128768e0b8: Log is configured to *not* fsync() on all Append() calls
I20260812 07:07:15.334375 23936 tablet_bootstrap.cc:654] T 42abd31269694419a6624e582d5f333d P 4dacdbf4f80b4fe1bc0f0ce0af724579: Neither blocks nor log segments found. Creating new log.
I20260812 07:07:15.335311 23935 tablet_bootstrap.cc:492] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08: No bootstrap required, opened a new log
I20260812 07:07:15.335598 23935 ts_tablet_manager.cc:1397] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260812 07:07:15.335743 23936 log.cc:824] T 42abd31269694419a6624e582d5f333d P 4dacdbf4f80b4fe1bc0f0ce0af724579: Log is configured to *not* fsync() on all Append() calls
I20260812 07:07:15.336516 23937 tablet_bootstrap.cc:492] T 42abd31269694419a6624e582d5f333d P 74c1eb79d71c44a5b1de64128768e0b8: No bootstrap required, opened a new log
I20260812 07:07:15.336794 23937 ts_tablet_manager.cc:1397] T 42abd31269694419a6624e582d5f333d P 74c1eb79d71c44a5b1de64128768e0b8: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260812 07:07:15.338867 23936 tablet_bootstrap.cc:492] T 42abd31269694419a6624e582d5f333d P 4dacdbf4f80b4fe1bc0f0ce0af724579: No bootstrap required, opened a new log
I20260812 07:07:15.339123 23936 ts_tablet_manager.cc:1397] T 42abd31269694419a6624e582d5f333d P 4dacdbf4f80b4fe1bc0f0ce0af724579: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260812 07:07:15.343492 23935 raft_consensus.cc:348] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c1eb79d71c44a5b1de64128768e0b8" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 42331 } } peers { permanent_uuid: "c01e069207904975a3e673740a311e08" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 36915 } } peers { permanent_uuid: "4dacdbf4f80b4fe1bc0f0ce0af724579" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38069 } }
I20260812 07:07:15.343899 23935 raft_consensus.cc:374] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:07:15.344007 23935 raft_consensus.cc:729] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c01e069207904975a3e673740a311e08, State: Initialized, Role: FOLLOWER
I20260812 07:07:15.344650 23935 consensus_queue.cc:260] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c1eb79d71c44a5b1de64128768e0b8" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 42331 } } peers { permanent_uuid: "c01e069207904975a3e673740a311e08" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 36915 } } peers { permanent_uuid: "4dacdbf4f80b4fe1bc0f0ce0af724579" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38069 } }
I20260812 07:07:15.346529 23935 ts_tablet_manager.cc:1428] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260812 07:07:15.347769 23937 raft_consensus.cc:348] T 42abd31269694419a6624e582d5f333d P 74c1eb79d71c44a5b1de64128768e0b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c1eb79d71c44a5b1de64128768e0b8" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 42331 } } peers { permanent_uuid: "c01e069207904975a3e673740a311e08" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 36915 } } peers { permanent_uuid: "4dacdbf4f80b4fe1bc0f0ce0af724579" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38069 } }
I20260812 07:07:15.348232 23937 raft_consensus.cc:374] T 42abd31269694419a6624e582d5f333d P 74c1eb79d71c44a5b1de64128768e0b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:07:15.348351 23937 raft_consensus.cc:729] T 42abd31269694419a6624e582d5f333d P 74c1eb79d71c44a5b1de64128768e0b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c1eb79d71c44a5b1de64128768e0b8, State: Initialized, Role: FOLLOWER
I20260812 07:07:15.349069 23937 consensus_queue.cc:260] T 42abd31269694419a6624e582d5f333d P 74c1eb79d71c44a5b1de64128768e0b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c1eb79d71c44a5b1de64128768e0b8" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 42331 } } peers { permanent_uuid: "c01e069207904975a3e673740a311e08" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 36915 } } peers { permanent_uuid: "4dacdbf4f80b4fe1bc0f0ce0af724579" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38069 } }
I20260812 07:07:15.352049 23936 raft_consensus.cc:348] T 42abd31269694419a6624e582d5f333d P 4dacdbf4f80b4fe1bc0f0ce0af724579 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c1eb79d71c44a5b1de64128768e0b8" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 42331 } } peers { permanent_uuid: "c01e069207904975a3e673740a311e08" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 36915 } } peers { permanent_uuid: "4dacdbf4f80b4fe1bc0f0ce0af724579" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38069 } }
I20260812 07:07:15.352715 23936 raft_consensus.cc:374] T 42abd31269694419a6624e582d5f333d P 4dacdbf4f80b4fe1bc0f0ce0af724579 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:07:15.352876 23936 raft_consensus.cc:729] T 42abd31269694419a6624e582d5f333d P 4dacdbf4f80b4fe1bc0f0ce0af724579 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dacdbf4f80b4fe1bc0f0ce0af724579, State: Initialized, Role: FOLLOWER
I20260812 07:07:15.352962 23762 heartbeater.cc:503] Master 127.6.61.62:43861 was elected leader, sending a full tablet report...
I20260812 07:07:15.353716 23937 ts_tablet_manager.cc:1428] T 42abd31269694419a6624e582d5f333d P 74c1eb79d71c44a5b1de64128768e0b8: Time spent starting tablet: real 0.017s user 0.009s sys 0.009s
I20260812 07:07:15.353538 23936 consensus_queue.cc:260] T 42abd31269694419a6624e582d5f333d P 4dacdbf4f80b4fe1bc0f0ce0af724579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c1eb79d71c44a5b1de64128768e0b8" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 42331 } } peers { permanent_uuid: "c01e069207904975a3e673740a311e08" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 36915 } } peers { permanent_uuid: "4dacdbf4f80b4fe1bc0f0ce0af724579" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38069 } }
I20260812 07:07:15.355304 23936 ts_tablet_manager.cc:1428] T 42abd31269694419a6624e582d5f333d P 4dacdbf4f80b4fe1bc0f0ce0af724579: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
I20260812 07:07:15.355701 23916 heartbeater.cc:503] Master 127.6.61.62:43861 was elected leader, sending a full tablet report...
W20260812 07:07:15.457379 23917 tablet.cc:2369] T 42abd31269694419a6624e582d5f333d P 4dacdbf4f80b4fe1bc0f0ce0af724579: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:07:15.467433 23941 raft_consensus.cc:482] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:07:15.467792 23941 raft_consensus.cc:504] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c1eb79d71c44a5b1de64128768e0b8" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 42331 } } peers { permanent_uuid: "c01e069207904975a3e673740a311e08" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 36915 } } peers { permanent_uuid: "4dacdbf4f80b4fe1bc0f0ce0af724579" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38069 } }
I20260812 07:07:15.469619 23941 leader_election.cc:288] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74c1eb79d71c44a5b1de64128768e0b8 (127.6.61.2:42331), 4dacdbf4f80b4fe1bc0f0ce0af724579 (127.6.61.3:38069)
I20260812 07:07:15.478441 23931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:15.470132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:42719 (local address 127.6.61.3:38069)
0812 07:07:15.470358 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:15.470367 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:15.471176 (+ 809us) server_negotiation.cc:407] Connection header received
0812 07:07:15.471455 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:15.471460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:15.471544 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:15.471646 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:07:15.472336 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.473324 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.476029 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.477290 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.477431 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:15.477545 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:15.477993 (+ 448us) server_negotiation.cc:299] Negotiation successful
0812 07:07:15.478163 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 07:07:15.478593 23944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:15.470221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:38069 (local address 127.6.61.1:42719)
0812 07:07:15.471028 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:15.471046 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:15.471265 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:15.471673 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:15.471678 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:15.471776 (+ 98us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:07:15.472185 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:15.472196 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.473479 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.473483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:15.475868 (+ 2385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:15.475885 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.477553 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.477562 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:15.477752 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:15.477835 (+ 83us) client_negotiation.cc:769] Sending connection context
0812 07:07:15.477936 (+ 101us) client_negotiation.cc:240] Negotiation successful
0812 07:07:15.478038 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":1,"spinlock_wait_cycles":23936,"thread_start_us":470,"threads_started":1}
I20260812 07:07:15.478710 23945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:15.470221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:42331 (local address 127.6.61.1:40149)
0812 07:07:15.471537 (+ 1316us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:15.471553 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:15.471718 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:15.472344 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:15.472347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:15.472366 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:07:15.472652 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:15.472658 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.473954 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.473960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:15.476075 (+ 2115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:15.476085 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.477751 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.477757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:15.478003 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:15.478036 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 07:07:15.478150 (+ 114us) client_negotiation.cc:240] Negotiation successful
0812 07:07:15.478283 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1210,"mutex_wait_us":29,"thread_start_us":946,"threads_started":1}
I20260812 07:07:15.478819 23928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:15.470227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:40149 (local address 127.6.61.2:42331)
0812 07:07:15.470432 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:15.470437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:15.471671 (+ 1234us) server_negotiation.cc:407] Connection header received
0812 07:07:15.472008 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:15.472015 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:15.472088 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:15.472173 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:07:15.472834 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.473754 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.476227 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.477514 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.477660 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:15.477764 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:15.478246 (+ 482us) server_negotiation.cc:299] Negotiation successful
0812 07:07:15.478393 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260812 07:07:15.479240 23764 tablet.cc:2369] T 42abd31269694419a6624e582d5f333d P 74c1eb79d71c44a5b1de64128768e0b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:07:15.479995 23861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42abd31269694419a6624e582d5f333d" candidate_uuid: "c01e069207904975a3e673740a311e08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dacdbf4f80b4fe1bc0f0ce0af724579" is_pre_election: true
I20260812 07:07:15.480365 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42abd31269694419a6624e582d5f333d" candidate_uuid: "c01e069207904975a3e673740a311e08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c1eb79d71c44a5b1de64128768e0b8" is_pre_election: true
I20260812 07:07:15.480654 23861 raft_consensus.cc:2454] T 42abd31269694419a6624e582d5f333d P 4dacdbf4f80b4fe1bc0f0ce0af724579 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c01e069207904975a3e673740a311e08 in term 0.
I20260812 07:07:15.480870 23711 raft_consensus.cc:2454] T 42abd31269694419a6624e582d5f333d P 74c1eb79d71c44a5b1de64128768e0b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c01e069207904975a3e673740a311e08 in term 0.
I20260812 07:07:15.481462 23475 leader_election.cc:302] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4dacdbf4f80b4fe1bc0f0ce0af724579, 74c1eb79d71c44a5b1de64128768e0b8, c01e069207904975a3e673740a311e08; no voters:
I20260812 07:07:15.481978 23941 raft_consensus.cc:2781] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:07:15.482147 23941 raft_consensus.cc:482] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:07:15.482270 23941 raft_consensus.cc:3037] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:07:15.484818 23941 raft_consensus.cc:504] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c1eb79d71c44a5b1de64128768e0b8" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 42331 } } peers { permanent_uuid: "c01e069207904975a3e673740a311e08" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 36915 } } peers { permanent_uuid: "4dacdbf4f80b4fe1bc0f0ce0af724579" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38069 } }
I20260812 07:07:15.485466 23941 leader_election.cc:288] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08 [CANDIDATE]: Term 1 election: Requested vote from peers 74c1eb79d71c44a5b1de64128768e0b8 (127.6.61.2:42331), 4dacdbf4f80b4fe1bc0f0ce0af724579 (127.6.61.3:38069)
I20260812 07:07:15.485986 23861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42abd31269694419a6624e582d5f333d" candidate_uuid: "c01e069207904975a3e673740a311e08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dacdbf4f80b4fe1bc0f0ce0af724579"
I20260812 07:07:15.485989 23711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42abd31269694419a6624e582d5f333d" candidate_uuid: "c01e069207904975a3e673740a311e08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c1eb79d71c44a5b1de64128768e0b8"
I20260812 07:07:15.486162 23861 raft_consensus.cc:3037] T 42abd31269694419a6624e582d5f333d P 4dacdbf4f80b4fe1bc0f0ce0af724579 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:07:15.486205 23711 raft_consensus.cc:3037] T 42abd31269694419a6624e582d5f333d P 74c1eb79d71c44a5b1de64128768e0b8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:07:15.488426 23861 raft_consensus.cc:2454] T 42abd31269694419a6624e582d5f333d P 4dacdbf4f80b4fe1bc0f0ce0af724579 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c01e069207904975a3e673740a311e08 in term 1.
I20260812 07:07:15.488601 23711 raft_consensus.cc:2454] T 42abd31269694419a6624e582d5f333d P 74c1eb79d71c44a5b1de64128768e0b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c01e069207904975a3e673740a311e08 in term 1.
I20260812 07:07:15.488906 23474 leader_election.cc:302] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dacdbf4f80b4fe1bc0f0ce0af724579, c01e069207904975a3e673740a311e08; no voters:
I20260812 07:07:15.489216 23941 raft_consensus.cc:2781] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:07:15.489732 23941 raft_consensus.cc:686] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08 [term 1 LEADER]: Becoming Leader. State: Replica: c01e069207904975a3e673740a311e08, State: Running, Role: LEADER
I20260812 07:07:15.490299 23941 consensus_queue.cc:237] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c1eb79d71c44a5b1de64128768e0b8" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 42331 } } peers { permanent_uuid: "c01e069207904975a3e673740a311e08" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 36915 } } peers { permanent_uuid: "4dacdbf4f80b4fe1bc0f0ce0af724579" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38069 } }
W20260812 07:07:15.494585 23607 tablet.cc:2369] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:07:15.495291 23385 catalog_manager.cc:5495] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08 reported cstate change: term changed from 0 to 1, leader changed from <none> to c01e069207904975a3e673740a311e08 (127.6.61.1). New cstate: current_term: 1 leader_uuid: "c01e069207904975a3e673740a311e08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c1eb79d71c44a5b1de64128768e0b8" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 42331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c01e069207904975a3e673740a311e08" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 36915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4dacdbf4f80b4fe1bc0f0ce0af724579" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38069 } health_report { overall_health: UNKNOWN } } }
I20260812 07:07:15.552436 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:15.546731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55078 (local address 127.6.61.1:36915)
0812 07:07:15.546963 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:15.546969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:15.546990 (+ 21us) server_negotiation.cc:407] Connection header received
0812 07:07:15.547108 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:15.547112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:15.547200 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:15.547370 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:07:15.547774 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.548596 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.550273 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.550684 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.550830 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:15.551334 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:07:15.551832 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:07:15.552011 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:15.552069 (+ 58us) server_negotiation.cc:299] Negotiation successful
0812 07:07:15.552162 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 07:07:15.562618 23711 raft_consensus.cc:1261] T 42abd31269694419a6624e582d5f333d P 74c1eb79d71c44a5b1de64128768e0b8 [term 1 FOLLOWER]: Refusing update from remote peer c01e069207904975a3e673740a311e08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:07:15.562618 23861 raft_consensus.cc:1261] T 42abd31269694419a6624e582d5f333d P 4dacdbf4f80b4fe1bc0f0ce0af724579 [term 1 FOLLOWER]: Refusing update from remote peer c01e069207904975a3e673740a311e08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:07:15.563997 23941 consensus_queue.cc:1035] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dacdbf4f80b4fe1bc0f0ce0af724579" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 38069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:07:15.564406 23946 consensus_queue.cc:1035] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74c1eb79d71c44a5b1de64128768e0b8" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 42331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:07:15.577589 23948 mvcc.cc:205] Tried to move back new op lower bound from 7317579512051408896 to 7317579511772217344. Current Snapshot: MvccSnapshot[applied={T|T < 7317579512051408896}]
I20260812 07:07:15.591318 23385 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:49924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:07:15.591642 23385 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:49924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:07:15.595701 23385 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04: Sending DeleteTablet for 3 replicas of tablet 42abd31269694419a6624e582d5f333d
I20260812 07:07:15.597039 23531 tablet_service.cc:1514] Processing DeleteTablet for tablet 42abd31269694419a6624e582d5f333d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:07:15 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:55066
I20260812 07:07:15.597039 23687 tablet_service.cc:1514] Processing DeleteTablet for tablet 42abd31269694419a6624e582d5f333d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:07:15 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:42214
I20260812 07:07:15.597769 23835 tablet_service.cc:1514] Processing DeleteTablet for tablet 42abd31269694419a6624e582d5f333d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:07:15 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:36808
I20260812 07:07:15.598752 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 23430
I20260812 07:07:15.607926 23960 tablet_replica.cc:330] T 42abd31269694419a6624e582d5f333d P 4dacdbf4f80b4fe1bc0f0ce0af724579: stopping tablet replica
I20260812 07:07:15.609208 23960 raft_consensus.cc:2229] T 42abd31269694419a6624e582d5f333d P 4dacdbf4f80b4fe1bc0f0ce0af724579 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:07:15.610656 23960 raft_consensus.cc:2258] T 42abd31269694419a6624e582d5f333d P 4dacdbf4f80b4fe1bc0f0ce0af724579 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:07:15.615139 23959 tablet_replica.cc:330] T 42abd31269694419a6624e582d5f333d P 74c1eb79d71c44a5b1de64128768e0b8: stopping tablet replica
I20260812 07:07:15.615847 23959 raft_consensus.cc:2229] T 42abd31269694419a6624e582d5f333d P 74c1eb79d71c44a5b1de64128768e0b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:07:15.616547 23959 raft_consensus.cc:2258] T 42abd31269694419a6624e582d5f333d P 74c1eb79d71c44a5b1de64128768e0b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:07:15.619391 23959 ts_tablet_manager.cc:1905] T 42abd31269694419a6624e582d5f333d P 74c1eb79d71c44a5b1de64128768e0b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:07:15.620657 23958 tablet_replica.cc:330] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08: stopping tablet replica
I20260812 07:07:15.621100 23958 raft_consensus.cc:2229] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08 [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:07:15.621984 23958 raft_consensus.cc:2258] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:07:15.624019 23960 ts_tablet_manager.cc:1905] T 42abd31269694419a6624e582d5f333d P 4dacdbf4f80b4fe1bc0f0ce0af724579: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:07:15.624027 23958 ts_tablet_manager.cc:1905] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:07:15.627894 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:15.599633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55086 (local address 127.6.61.1:36915)
0812 07:07:15.600455 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:15.600465 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:15.600488 (+ 23us) server_negotiation.cc:407] Connection header received
0812 07:07:15.600564 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:15.600572 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:15.600672 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:15.600803 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:15.603644 (+ 2841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.605447 (+ 1803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.608458 (+ 3011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.609320 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.609471 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:15.615192 (+ 5721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:15.615213 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:15.615239 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:15.615305 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:15.620351 (+ 5046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:15.621535 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:15.621548 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:15.621552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:15.621768 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:15.624696 (+ 2928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:15.624706 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:15.624713 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:15.625279 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:15.625629 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:15.627120 (+ 1491us) server_negotiation.cc:299] Negotiation successful
0812 07:07:15.627428 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647}
I20260812 07:07:15.628542 23959 ts_tablet_manager.cc:1918] T 42abd31269694419a6624e582d5f333d P 74c1eb79d71c44a5b1de64128768e0b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:07:15.628808 23959 log.cc:1196] T 42abd31269694419a6624e582d5f333d P 74c1eb79d71c44a5b1de64128768e0b8: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/wal/wals/42abd31269694419a6624e582d5f333d
I20260812 07:07:15.629101 23599 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:07:15.629606 23959 ts_tablet_manager.cc:1939] T 42abd31269694419a6624e582d5f333d P 74c1eb79d71c44a5b1de64128768e0b8: Deleting consensus metadata
I20260812 07:07:15.633414 23370 catalog_manager.cc:4841] TS 74c1eb79d71c44a5b1de64128768e0b8 (127.6.61.2:42331): tablet 42abd31269694419a6624e582d5f333d (table test-table [id=372f4e65677247079f7a2da16a53abed]) successfully deleted
I20260812 07:07:15.637756 23960 ts_tablet_manager.cc:1918] T 42abd31269694419a6624e582d5f333d P 4dacdbf4f80b4fe1bc0f0ce0af724579: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:07:15.638031 23960 log.cc:1196] T 42abd31269694419a6624e582d5f333d P 4dacdbf4f80b4fe1bc0f0ce0af724579: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/wal/wals/42abd31269694419a6624e582d5f333d
I20260812 07:07:15.638661 23960 ts_tablet_manager.cc:1939] T 42abd31269694419a6624e582d5f333d P 4dacdbf4f80b4fe1bc0f0ce0af724579: Deleting consensus metadata
I20260812 07:07:15.640843 23369 catalog_manager.cc:4841] TS 4dacdbf4f80b4fe1bc0f0ce0af724579 (127.6.61.3:38069): tablet 42abd31269694419a6624e582d5f333d (table test-table [id=372f4e65677247079f7a2da16a53abed]) successfully deleted
I20260812 07:07:15.742748 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 23430
W20260812 07:07:15.770576 23372 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.61.1:36915 (error 108)
I20260812 07:07:15.770859 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 23611
I20260812 07:07:15.772861 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:15.772168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:36915 (local address 127.0.0.1:55094)
0812 07:07:15.772536 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:15.772675 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260812 07:07:15.773134 23372 catalog_manager.cc:4584] TS c01e069207904975a3e673740a311e08 (127.6.61.1:36915): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 42abd31269694419a6624e582d5f333d: Network error: Client connection negotiation failed: client connection to 127.6.61.1:36915: connect: Connection refused (error 111)
I20260812 07:07:15.785315 23928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:15.771725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42228 (local address 127.6.61.2:42331)
0812 07:07:15.771896 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:15.771900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:15.771919 (+ 19us) server_negotiation.cc:407] Connection header received
0812 07:07:15.772091 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:15.772097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:15.772185 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:15.772298 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:15.774012 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.774808 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.776217 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.776615 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.776739 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:15.779582 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:15.779595 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:15.779610 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:15.779650 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:15.782554 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:15.783254 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:15.783261 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:15.783264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:15.783384 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:15.783821 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:15.783825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:15.783828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:15.784269 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:15.784492 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:15.784838 (+ 346us) server_negotiation.cc:299] Negotiation successful
0812 07:07:15.785007 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 07:07:15.786161 23756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:07:15.827000 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:15.826443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:36915 (local address 127.0.0.1:55100)
0812 07:07:15.826695 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:15.826813 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260812 07:07:15.828163 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:15.827757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:36915 (local address 127.0.0.1:55110)
0812 07:07:15.827917 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:15.828006 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260812 07:07:15.877800 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 23611
I20260812 07:07:15.888809 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:15.888148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:36915 (local address 127.0.0.1:55120)
0812 07:07:15.888427 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:15.888556 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260812 07:07:15.890158 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:15.889652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:36915 (local address 127.0.0.1:55136)
0812 07:07:15.889882 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:15.889986 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260812 07:07:15.899672 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 23766
I20260812 07:07:15.914083 23931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:15.900487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36812 (local address 127.6.61.3:38069)
0812 07:07:15.900652 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:15.900657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:15.900675 (+ 18us) server_negotiation.cc:407] Connection header received
0812 07:07:15.900879 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:15.900884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:15.900968 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:15.901062 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:15.902588 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.903323 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.904722 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:15.905111 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:15.905209 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:15.908215 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:15.908233 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:15.908252 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:15.908304 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:15.911104 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:15.911840 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:15.911848 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:15.911850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:15.911993 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:15.912517 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:15.912521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:15.912524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:15.912900 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:15.913134 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:15.913544 (+ 410us) server_negotiation.cc:299] Negotiation successful
0812 07:07:15.913742 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 07:07:15.914744 23911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:07:15.971086 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:15.970440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:36915 (local address 127.0.0.1:55140)
0812 07:07:15.970712 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:15.970851 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260812 07:07:15.972359 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:15.971885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:36915 (local address 127.0.0.1:55142)
0812 07:07:15.972121 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:15.972219 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260812 07:07:16.006434 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 23766
I20260812 07:07:16.028580 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 23350
I20260812 07:07:16.030071 23422 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:07:16.163069 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:16.162417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:36915 (local address 127.0.0.1:55150)
0812 07:07:16.162655 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:16.162794 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:36915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260812 07:07:16.163460 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 23350
I20260812 07:07:16.191521 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:43861
--webserver_interface=127.6.61.62
--webserver_port=36017
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:45733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:43861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:07:16.538347 23967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:07:16.538666 23967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:07:16.538821 23967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:07:16.546224 23967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:07:16.546353 23967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:07:16.546446 23967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:07:16.546541 23967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:07:16.546608 23967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:07:16.556650 23967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:45733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:43861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:43861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=36017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:07:16.558115 23967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:07:16.559489 23967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:07:16.571471 23973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:16.571513 23974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:16.571543 23976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:07:16.572237 23967 server_base.cc:1034] running on GCE node
I20260812 07:07:16.573021 23967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:07:16.574509 23967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:07:16.575742 23967 hybrid_clock.cc:648] HybridClock initialized: now 1786518436575686 us; error 68 us; skew 500 ppm
I20260812 07:07:16.576095 23967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:07:16 dist-test-slave-69rg krb5kdc[22702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518436, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:07:16.579413 23967 init.cc:377] Logged in from keytab as kudu/127.6.61.62@KRBTEST.COM (short username kudu)
I20260812 07:07:16.580996 23967 webserver.cc:460] Webserver started at http://127.6.61.62:36017/ using document root <none> and password file <none>
I20260812 07:07:16.581705 23967 fs_manager.cc:359] Metadata directory not provided
I20260812 07:07:16.581880 23967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:07:16.587426 23967 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260812 07:07:16.590525 23985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:16.591835 23967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:07:16.592024 23967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "de11599be8e84615bc177368a6e7fb04"
format_stamp: "Formatted at 2026-08-12 07:07:10 on dist-test-slave-69rg"
server_key: "c35c551071f6d503fabb514bea3aaba4"
server_key_iv: "6d74cb59effa8d5cf2cf2841f2d1c9f5"
server_key_version: "encryptionkey@0"
I20260812 07:07:16.593034 23967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:07:16.605173 23967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:07:16.609993 23967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:07:16.610313 23967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:07:16.630658 23967 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:43861
I20260812 07:07:16.630702 24048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:43861 every 8 connection(s)
I20260812 07:07:16.632285 23967 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:07:16.639629 24049 sys_catalog.cc:263] Verifying existing consensus state
I20260812 07:07:16.641865 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 23967
I20260812 07:07:16.642978 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.1:36915
--local_ip_for_outbound_sockets=127.6.61.1
--tserver_master_addrs=127.6.61.62:43861
--webserver_port=46173
--webserver_interface=127.6.61.1
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:45733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:07:16.643999 24049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04: Bootstrap starting.
I20260812 07:07:16.668779 24049 log.cc:824] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04: Log is configured to *not* fsync() on all Append() calls
I20260812 07:07:16.685992 24049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 07:07:16.686573 24049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04: Bootstrap complete.
I20260812 07:07:16.696012 24049 raft_consensus.cc:348] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de11599be8e84615bc177368a6e7fb04" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 43861 } }
I20260812 07:07:16.697105 24049 raft_consensus.cc:729] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: de11599be8e84615bc177368a6e7fb04, State: Initialized, Role: FOLLOWER
I20260812 07:07:16.697788 24049 consensus_queue.cc:260] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de11599be8e84615bc177368a6e7fb04" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 43861 } }
I20260812 07:07:16.698155 24049 raft_consensus.cc:388] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:07:16.698318 24049 raft_consensus.cc:482] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:07:16.698520 24049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [term 2 FOLLOWER]: Advancing to term 3
I20260812 07:07:16.702728 24049 raft_consensus.cc:504] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de11599be8e84615bc177368a6e7fb04" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 43861 } }
I20260812 07:07:16.703265 24049 leader_election.cc:302] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de11599be8e84615bc177368a6e7fb04; no voters:
I20260812 07:07:16.704011 24049 leader_election.cc:288] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 07:07:16.704161 24053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 07:07:16.705629 24053 raft_consensus.cc:686] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [term 3 LEADER]: Becoming Leader. State: Replica: de11599be8e84615bc177368a6e7fb04, State: Running, Role: LEADER
I20260812 07:07:16.706326 24053 consensus_queue.cc:237] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de11599be8e84615bc177368a6e7fb04" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 43861 } }
I20260812 07:07:16.706579 24049 sys_catalog.cc:564] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:07:16.709038 24055 sys_catalog.cc:455] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de11599be8e84615bc177368a6e7fb04. Latest consensus state: current_term: 3 leader_uuid: "de11599be8e84615bc177368a6e7fb04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de11599be8e84615bc177368a6e7fb04" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 43861 } } }
I20260812 07:07:16.709465 24055 sys_catalog.cc:458] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [sys.catalog]: This master's current role is: LEADER
I20260812 07:07:16.709800 24054 sys_catalog.cc:455] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "de11599be8e84615bc177368a6e7fb04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de11599be8e84615bc177368a6e7fb04" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 43861 } } }
I20260812 07:07:16.710187 24054 sys_catalog.cc:458] T 00000000000000000000000000000000 P de11599be8e84615bc177368a6e7fb04 [sys.catalog]: This master's current role is: LEADER
I20260812 07:07:16.719923 24064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:07:16.729440 24064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:07:16.731102 24064 catalog_manager.cc:1259] Loaded cluster ID: 137db745a3ca4d959d8b8dc6c9f5b847
I20260812 07:07:16.731282 24064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260812 07:07:16.733628 24064 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f8e952869ad google::LogMessage::Fail() at ??:0
@ 0x7f8e9528988a google::LogMessage::SendToLog() at ??:0
@ 0x7f8e952864de google::LogMessage::Flush() at ??:0
@ 0x7f8e95287a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8eaacb9512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f8eaacb6976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f8eaad3e610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f8eaad3e439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f8e97ed2345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f8e97ed591c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f8e97ed5799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f8e97eb0eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8e9b18a6db start_thread at ??:0
@ 0x7f8e9412061f clone at ??:0
W20260812 07:07:17.058391 24051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:07:17.058712 24051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:07:17.058881 24051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:07:17.066529 24051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:07:17.066658 24051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:07:17.066845 24051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.1
I20260812 07:07:17.077200 24051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:45733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.1:36915
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.61.1
--webserver_port=46173
--tserver_master_addrs=127.6.61.62:43861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.1
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:07:17.078672 24051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:07:17.080052 24051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:07:17.094906 24094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:17.095017 24098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:17.095053 24095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:07:17.096342 24051 server_base.cc:1034] running on GCE node
I20260812 07:07:17.097039 24051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:07:17.098094 24051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:07:17.099423 24051 hybrid_clock.cc:648] HybridClock initialized: now 1786518437099372 us; error 36 us; skew 500 ppm
I20260812 07:07:17.099754 24051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:07:17 dist-test-slave-69rg krb5kdc[22702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518437, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:07:17.109362 24051 init.cc:377] Logged in from keytab as kudu/127.6.61.1@KRBTEST.COM (short username kudu)
I20260812 07:07:17.111243 24051 webserver.cc:460] Webserver started at http://127.6.61.1:46173/ using document root <none> and password file <none>
I20260812 07:07:17.112027 24051 fs_manager.cc:359] Metadata directory not provided
I20260812 07:07:17.112164 24051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:07:17.118019 24051 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 07:07:17.121026 24105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:17.122534 24051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:07:17.122720 24051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/wal
uuid: "c01e069207904975a3e673740a311e08"
format_stamp: "Formatted at 2026-08-12 07:07:11 on dist-test-slave-69rg"
server_key: "aec759c06f64446dd7d3082a926807da"
server_key_iv: "58116310d115f420baf254d11a6a550b"
server_key_version: "encryptionkey@0"
I20260812 07:07:17.123734 24051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:07:17.152165 24051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:07:17.156960 24051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:07:17.157262 24051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:07:17.158461 24051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:07:17.160385 24113 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260812 07:07:17.165691 24113 ts_tablet_manager.cc:1852] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260812 07:07:17.166802 24113 ts_tablet_manager.cc:1905] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:07:17.174293 24113 ts_tablet_manager.cc:1918] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:07:17.174475 24113 log.cc:1196] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/wal/wals/42abd31269694419a6624e582d5f333d
I20260812 07:07:17.174918 24113 ts_tablet_manager.cc:1939] T 42abd31269694419a6624e582d5f333d P c01e069207904975a3e673740a311e08: Deleting consensus metadata
I20260812 07:07:17.175310 24051 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260812 07:07:17.175470 24051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.000s sys 0.001s
I20260812 07:07:17.175642 24051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:07:17.175745 24051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:17.212136 24051 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.1:36915
I20260812 07:07:17.212214 24235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.1:36915 every 8 connection(s)
I20260812 07:07:17.213594 24051 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
I20260812 07:07:17.218940 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 24051
I20260812 07:07:17.219107 24239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:17.216771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.1:55821)
0812 07:07:17.217551 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:17.217734 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":326,"threads_started":1}
I20260812 07:07:17.219830 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.2:42331
--local_ip_for_outbound_sockets=127.6.61.2
--tserver_master_addrs=127.6.61.62:43861
--webserver_port=39753
--webserver_interface=127.6.61.2
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:45733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:07:17.219866 24236 heartbeater.cc:644] Failed to heartbeat to 127.6.61.62:43861 (0 consecutive failures): Network error: Failed to ping master at 127.6.61.62:43861: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
I20260812 07:07:17.220842 24239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:17.220399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.1:59829)
0812 07:07:17.220591 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:17.220686 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260812 07:07:17.221886 24239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:17.221401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.1:40799)
0812 07:07:17.221571 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:17.221673 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 07:07:17.222191 24236 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 07:07:17.572775 24240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:07:17.573066 24240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:07:17.573246 24240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:07:17.580878 24240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:07:17.580997 24240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:07:17.581218 24240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.2
I20260812 07:07:17.591145 24240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:45733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.2:42331
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.61.2
--webserver_port=39753
--tserver_master_addrs=127.6.61.62:43861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.2
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:07:17.592531 24240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:07:17.593873 24240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:07:17.607407 24247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:17.607369 24246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:17.607586 24250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:07:17.608558 24240 server_base.cc:1034] running on GCE node
I20260812 07:07:17.609277 24240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:07:17.610404 24240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:07:17.611658 24240 hybrid_clock.cc:648] HybridClock initialized: now 1786518437611594 us; error 46 us; skew 500 ppm
I20260812 07:07:17.612026 24240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:07:17 dist-test-slave-69rg krb5kdc[22702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518437, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:07:17.615806 24240 init.cc:377] Logged in from keytab as kudu/127.6.61.2@KRBTEST.COM (short username kudu)
I20260812 07:07:17.617446 24240 webserver.cc:460] Webserver started at http://127.6.61.2:39753/ using document root <none> and password file <none>
I20260812 07:07:17.618345 24240 fs_manager.cc:359] Metadata directory not provided
I20260812 07:07:17.618515 24240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:07:17.624907 24240 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260812 07:07:17.628163 24258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:17.629719 24240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 07:07:17.629973 24240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/wal
uuid: "74c1eb79d71c44a5b1de64128768e0b8"
format_stamp: "Formatted at 2026-08-12 07:07:11 on dist-test-slave-69rg"
server_key: "00bcc27a935bc7bc906722eeeb848a12"
server_key_iv: "b5df1704f6660f6017915cf24c0b7c27"
server_key_version: "encryptionkey@0"
I20260812 07:07:17.631000 24240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:07:17.645673 24240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:07:17.650007 24240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:07:17.650367 24240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:07:17.651383 24240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:07:17.652733 24240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:07:17.652855 24240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:17.652933 24240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:07:17.652990 24240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:17.687523 24240 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.2:42331
I20260812 07:07:17.687630 24391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.2:42331 every 8 connection(s)
I20260812 07:07:17.689175 24240 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
I20260812 07:07:17.691440 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 24240
I20260812 07:07:17.692078 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.3:38069
--local_ip_for_outbound_sockets=127.6.61.3
--tserver_master_addrs=127.6.61.62:43861
--webserver_port=41439
--webserver_interface=127.6.61.3
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:45733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:07:17.694746 24394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:17.692564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.2:57201)
0812 07:07:17.693497 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:17.693749 (+ 252us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":377,"threads_started":1}
W20260812 07:07:17.695523 24392 heartbeater.cc:644] Failed to heartbeat to 127.6.61.62:43861 (0 consecutive failures): Network error: Failed to ping master at 127.6.61.62:43861: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
I20260812 07:07:17.696542 24394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:17.696173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.2:37881)
0812 07:07:17.696356 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:17.696429 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260812 07:07:17.697513 24394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:17.697145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.2:52733)
0812 07:07:17.697313 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:17.697398 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260812 07:07:17.697796 24392 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 07:07:18.038331 24395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:07:18.038611 24395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:07:18.038761 24395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:07:18.046157 24395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:07:18.046281 24395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:07:18.046497 24395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.3
I20260812 07:07:18.056260 24395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:45733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.3:38069
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.61.3
--webserver_port=41439
--tserver_master_addrs=127.6.61.62:43861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.3
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:07:18.057644 24395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:07:18.059036 24395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:07:18.074815 24402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:18.074883 24403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:18.074857 24405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:07:18.076109 24395 server_base.cc:1034] running on GCE node
I20260812 07:07:18.076740 24395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:07:18.077670 24395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:07:18.078877 24395 hybrid_clock.cc:648] HybridClock initialized: now 1786518438078824 us; error 63 us; skew 500 ppm
I20260812 07:07:18.079180 24395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:07:18 dist-test-slave-69rg krb5kdc[22702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518438, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:07:18.082491 24395 init.cc:377] Logged in from keytab as kudu/127.6.61.3@KRBTEST.COM (short username kudu)
I20260812 07:07:18.084051 24395 webserver.cc:460] Webserver started at http://127.6.61.3:41439/ using document root <none> and password file <none>
I20260812 07:07:18.084762 24395 fs_manager.cc:359] Metadata directory not provided
I20260812 07:07:18.084896 24395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:07:18.090410 24395 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260812 07:07:18.093266 24413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:18.094815 24395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:07:18.095007 24395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/wal
uuid: "4dacdbf4f80b4fe1bc0f0ce0af724579"
format_stamp: "Formatted at 2026-08-12 07:07:12 on dist-test-slave-69rg"
server_key: "6c8610a2d2256f713d77ec73c927d79c"
server_key_iv: "b3b74dd0b297ecd88ba2c248e43d9580"
server_key_version: "encryptionkey@0"
I20260812 07:07:18.096022 24395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:07:18.136763 24395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:07:18.141387 24395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:07:18.141690 24395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:07:18.142870 24395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:07:18.144357 24395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:07:18.144491 24395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:18.144637 24395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:07:18.144730 24395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:18.180156 24395 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.3:38069
I20260812 07:07:18.180258 24549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.3:38069 every 8 connection(s)
I20260812 07:07:18.181664 24395 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
I20260812 07:07:18.185513 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 24395
I20260812 07:07:18.187101 24553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:18.184942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.3:54003)
0812 07:07:18.185954 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:18.186230 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":344,"threads_started":1}
W20260812 07:07:18.187970 24551 heartbeater.cc:644] Failed to heartbeat to 127.6.61.62:43861 (0 consecutive failures): Network error: Failed to ping master at 127.6.61.62:43861: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
I20260812 07:07:18.188874 24553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:18.188529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.3:55797)
0812 07:07:18.188678 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:18.188768 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260812 07:07:18.189940 24553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:18.189475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.3:50699)
0812 07:07:18.189663 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:18.189746 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260812 07:07:18.190248 24551 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 07:07:18.223650 24555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:18.222773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.1:58301)
0812 07:07:18.223262 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:18.223387 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":237,"threads_started":1}
I20260812 07:07:18.699229 24556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:18.698448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.2:56491)
0812 07:07:18.698907 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:18.698989 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":218,"threads_started":1}
I20260812 07:07:19.191887 24558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:19.190898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.3:51775)
0812 07:07:19.191449 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:19.191565 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":267,"threads_started":1}
I20260812 07:07:19.225461 24559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:19.224729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.1:58631)
0812 07:07:19.225197 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:19.225288 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":251,"threads_started":1}
I20260812 07:07:19.701087 24560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:19.700239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.2:48447)
0812 07:07:19.700825 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:19.700916 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":332,"threads_started":1}
I20260812 07:07:20.194001 24561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:20.193009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.3:45245)
0812 07:07:20.193627 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:20.193729 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":353,"threads_started":1}
I20260812 07:07:20.227422 24562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:20.226444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.1:60473)
0812 07:07:20.227050 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:20.227145 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":320,"threads_started":1}
I20260812 07:07:20.702939 24563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:20.702193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.2:37615)
0812 07:07:20.702662 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:20.702765 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":252,"threads_started":1}
I20260812 07:07:21.195807 24564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:21.194995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.3:42373)
0812 07:07:21.195492 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:21.195582 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":261,"threads_started":1}
I20260812 07:07:21.229388 24565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:21.228494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.1:59497)
0812 07:07:21.229072 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:21.229180 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":341,"threads_started":1}
I20260812 07:07:21.704856 24566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:21.703851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.2:56749)
0812 07:07:21.704427 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:21.704549 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":288,"threads_started":1}
I20260812 07:07:22.197551 24568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:22.196793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.3:41249)
0812 07:07:22.197261 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:22.197373 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":241,"threads_started":1}
I20260812 07:07:22.231115 24569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:22.230368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.1:57419)
0812 07:07:22.230852 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:22.230935 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":243,"threads_started":1}
I20260812 07:07:22.706686 24571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:22.705915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.2:58713)
0812 07:07:22.706424 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:22.706511 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":242,"threads_started":1}
I20260812 07:07:23.199519 24572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:23.198673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.3:57437)
0812 07:07:23.199176 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:23.199335 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":278,"threads_started":1}
I20260812 07:07:23.232738 24573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:23.231987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.1:59889)
0812 07:07:23.232490 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:23.232570 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":294,"threads_started":1}
I20260812 07:07:23.708628 24574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:23.707804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.2:55437)
0812 07:07:23.708341 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:23.708448 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":257,"threads_started":1}
I20260812 07:07:24.201476 24575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:24.200605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.3:55533)
0812 07:07:24.201125 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:24.201279 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":271,"threads_started":1}
I20260812 07:07:24.234712 24577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:24.233646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.1:39853)
0812 07:07:24.234250 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:24.234353 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":308,"threads_started":1}
I20260812 07:07:24.710551 24578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:24.709682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.2:41631)
0812 07:07:24.710235 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:24.710322 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":274,"threads_started":1}
I20260812 07:07:25.203287 24579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:25.202451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.3:39347)
0812 07:07:25.202987 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:25.203067 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":275,"threads_started":1}
I20260812 07:07:25.236433 24580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:25.235745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.1:54401)
0812 07:07:25.236176 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:25.236258 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":220,"threads_started":1}
I20260812 07:07:25.712426 24581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:25.711593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.2:48273)
0812 07:07:25.712123 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:25.712211 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":296,"threads_started":1}
I20260812 07:07:26.205951 24582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:26.204596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.3:40701)
0812 07:07:26.205504 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:26.205648 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":525,"threads_started":1}
I20260812 07:07:26.238531 24583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:26.237561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.1:35891)
0812 07:07:26.238224 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:26.238338 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":353,"threads_started":1}
I20260812 07:07:26.714394 24584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:26.713473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.2:59331)
0812 07:07:26.714077 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:26.714207 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":274,"threads_started":1}
I20260812 07:07:27.207870 24585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:27.207063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.3:41145)
0812 07:07:27.207582 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:27.207669 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":249,"threads_started":1}
I20260812 07:07:27.240214 24586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:27.239518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.1:53781)
0812 07:07:27.239962 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:27.240043 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":245,"threads_started":1}
I20260812 07:07:27.716281 24587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:27.715514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.2:44725)
0812 07:07:27.716019 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:27.716108 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":276,"threads_started":1}
I20260812 07:07:28.209748 24588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:28.208962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.3:43657)
0812 07:07:28.209481 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:28.209575 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":275,"threads_started":1}
I20260812 07:07:28.242105 24589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:28.241240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.1:48089)
0812 07:07:28.241738 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:28.241893 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":279,"threads_started":1}
I20260812 07:07:28.718159 24591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:28.717336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.2:38319)
0812 07:07:28.717857 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:28.717951 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":237,"threads_started":1}
I20260812 07:07:29.211585 24592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:29.210835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.3:51543)
0812 07:07:29.211301 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:29.211399 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":228,"threads_started":1}
I20260812 07:07:29.243896 24593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:29.243113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.1:54605)
0812 07:07:29.243617 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:29.243709 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":259,"threads_started":1}
I20260812 07:07:29.719992 24594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:29.719234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.2:52569)
0812 07:07:29.719742 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:29.719830 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":267,"threads_started":1}
I20260812 07:07:30.213526 24596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:30.212702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.3:41467)
0812 07:07:30.213279 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:30.213370 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":211,"threads_started":1}
I20260812 07:07:30.245932 24597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:30.245007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.1:55665)
0812 07:07:30.245540 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:30.245625 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":295,"threads_started":1}
I20260812 07:07:30.721742 24598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:30.720966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.2:57641)
0812 07:07:30.721455 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:30.721541 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":227,"threads_started":1}
I20260812 07:07:31.215368 24599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:31.214562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.3:39219)
0812 07:07:31.215056 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:31.215143 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":257,"threads_started":1}
I20260812 07:07:31.247663 24600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:31.246927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.1:60655)
0812 07:07:31.247387 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:31.247472 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":223,"threads_started":1}
I20260812 07:07:31.723729 24601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:31.722899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.2:45583)
0812 07:07:31.723426 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:31.723517 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":283,"threads_started":1}
I20260812 07:07:32.217110 24602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:32.216336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.3:47669)
0812 07:07:32.216844 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:32.216925 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":266,"threads_started":1}
I20260812 07:07:32.249635 24603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:32.248806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.1:51853)
0812 07:07:32.249347 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:32.249428 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":299,"threads_started":1}
I20260812 07:07:32.726019 24604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:32.724976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.2:47681)
0812 07:07:32.725586 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:32.725711 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":318,"threads_started":1}
I20260812 07:07:33.187359 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 24051
I20260812 07:07:33.206677 24605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:33.188588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37918 (local address 127.6.61.1:36915)
0812 07:07:33.189242 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:33.189258 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:33.189298 (+ 40us) server_negotiation.cc:407] Connection header received
0812 07:07:33.191113 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:33.191124 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:33.191559 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:33.191870 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:33.193294 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:33.194707 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:33.196174 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:33.196873 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:33.197018 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:33.199960 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:33.199969 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:33.199986 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:33.200026 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:33.203322 (+ 3296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:33.204042 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:33.204049 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:33.204056 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:33.204242 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:33.204715 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:33.204719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:33.204722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:33.205292 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:33.205533 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:33.205975 (+ 442us) server_negotiation.cc:299] Negotiation successful
0812 07:07:33.206348 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":351,"threads_started":1}
I20260812 07:07:33.208367 24230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:07:33.219300 24609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:33.218437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.3:57705)
0812 07:07:33.219006 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:33.219104 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":317,"threads_started":1}
I20260812 07:07:33.298749 24611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:33.294077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:43861 (local address 127.6.61.1:59145)
0812 07:07:33.298318 (+ 4241us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:33.298469 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4091,"thread_start_us":3935,"threads_started":1}
I20260812 07:07:33.299221 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 24051
I20260812 07:07:33.322157 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 24240
I20260812 07:07:33.341528 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:33.323329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50176 (local address 127.6.61.2:42331)
0812 07:07:33.323898 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:33.323914 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:33.323952 (+ 38us) server_negotiation.cc:407] Connection header received
0812 07:07:33.325538 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:33.325549 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:33.326068 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:33.326370 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:33.328199 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:33.329484 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:33.331064 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:33.331790 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:33.332006 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:33.335134 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:33.335145 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:33.335162 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:33.335213 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:33.338532 (+ 3319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:33.339208 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:33.339215 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:33.339222 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:33.339380 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:33.339726 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:33.339729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:33.339731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:33.340266 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:33.340502 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:33.340894 (+ 392us) server_negotiation.cc:299] Negotiation successful
0812 07:07:33.341237 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":284,"threads_started":1}
I20260812 07:07:33.343040 24387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:07:33.428211 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 24240
I20260812 07:07:33.450990 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 24395
I20260812 07:07:33.470350 24614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:33.452018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55334 (local address 127.6.61.3:38069)
0812 07:07:33.452664 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:33.452678 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:33.452721 (+ 43us) server_negotiation.cc:407] Connection header received
0812 07:07:33.454468 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:33.454480 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:33.454912 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:33.455269 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:33.456848 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:33.458408 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:33.459711 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:33.460392 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:33.460592 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:33.463448 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:33.463457 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:33.463473 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:33.463524 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:33.466795 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:33.467536 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:33.467544 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:33.467551 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:33.467727 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:33.468135 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:33.468140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:33.468144 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:33.468701 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:33.469022 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:33.469518 (+ 496us) server_negotiation.cc:299] Negotiation successful
0812 07:07:33.470018 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":320,"threads_started":1}
I20260812 07:07:33.472059 24544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:07:33.558239 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 24395
2026-08-12T07:07:33Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25102 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:07:33 dist-test-slave-69rg krb5kdc[24621](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:07:33 dist-test-slave-69rg krb5kdc[24621](info): set up 2 sockets
Aug 12 07:07:33 dist-test-slave-69rg krb5kdc[24621](info): commencing operation
krb5kdc: starting...
W20260812 07:07:34.800117 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.197s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:07:34 dist-test-slave-69rg krb5kdc[24621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:07:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:07:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
I20260812 07:07:35.114229 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:39783
--webserver_interface=127.6.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:46191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:39783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:07:35.463013 24636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:07:35.463455 24636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:07:35.463620 24636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:07:35.471092 24636 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 07:07:35.471267 24636 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 07:07:35.471345 24636 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 07:07:35.471422 24636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:07:35.471529 24636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:07:35.471635 24636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:07:35.471732 24636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:07:35.471813 24636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:07:35.482277 24636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:46191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:39783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:39783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:07:35.483831 24636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:07:35.485220 24636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:07:35.499742 24642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:07:35.499828 24636 server_base.cc:1034] running on GCE node
W20260812 07:07:35.499738 24645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:35.499729 24643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:07:35.501117 24636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:07:35.502807 24636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:07:35.504041 24636 hybrid_clock.cc:648] HybridClock initialized: now 1786518455503959 us; error 51 us; skew 500 ppm
I20260812 07:07:35.504433 24636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:07:35 dist-test-slave-69rg krb5kdc[24621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518455, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:07:35.507872 24636 init.cc:377] Logged in from keytab as kudu/127.6.61.62@KRBTEST.COM (short username kudu)
I20260812 07:07:35.509464 24636 webserver.cc:460] Webserver started at http://127.6.61.62:33503/ using document root <none> and password file <none>
I20260812 07:07:35.510239 24636 fs_manager.cc:359] Metadata directory not provided
I20260812 07:07:35.510334 24636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:07:35.510596 24636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:07:35.513603 24636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "cf960fc0f93f47f4bbb12e9580b94392"
format_stamp: "Formatted at 2026-08-12 07:07:35 on dist-test-slave-69rg"
server_key: "bce82cc2ad292df4a17751b72e88e4cf"
server_key_iv: "43b5eac11b3a46d174730a9400f0ad99"
server_key_version: "encryptionkey@0"
I20260812 07:07:35.514477 24636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "cf960fc0f93f47f4bbb12e9580b94392"
format_stamp: "Formatted at 2026-08-12 07:07:35 on dist-test-slave-69rg"
server_key: "bce82cc2ad292df4a17751b72e88e4cf"
server_key_iv: "43b5eac11b3a46d174730a9400f0ad99"
server_key_version: "encryptionkey@0"
I20260812 07:07:35.519665 24636 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 07:07:35.523524 24655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:35.525002 24636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 07:07:35.525143 24636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "cf960fc0f93f47f4bbb12e9580b94392"
format_stamp: "Formatted at 2026-08-12 07:07:35 on dist-test-slave-69rg"
server_key: "bce82cc2ad292df4a17751b72e88e4cf"
server_key_iv: "43b5eac11b3a46d174730a9400f0ad99"
server_key_version: "encryptionkey@0"
I20260812 07:07:35.525334 24636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:07:35.548527 24636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:07:35.552882 24636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:07:35.553138 24636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:07:35.572677 24636 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:39783
I20260812 07:07:35.572698 24712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:39783 every 8 connection(s)
I20260812 07:07:35.574286 24636 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:07:35.575459 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 24636
I20260812 07:07:35.575735 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:07:35.576668 6388 external_mini_cluster.cc:1428] Setting key 96c206e8870307de8b5d7b9d04a2cee5
I20260812 07:07:35.579543 24713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:07:35.590674 24713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392: Bootstrap starting.
Aug 12 07:07:35 dist-test-slave-69rg krb5kdc[24621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:07:35.594573 24713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392: Neither blocks nor log segments found. Creating new log.
I20260812 07:07:35.595932 24713 log.cc:824] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392: Log is configured to *not* fsync() on all Append() calls
I20260812 07:07:35.599741 24713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392: No bootstrap required, opened a new log
I20260812 07:07:35.602411 24716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:35.579066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36358 (local address 127.6.61.62:39783)
0812 07:07:35.579928 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:35.579941 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:35.579984 (+ 43us) server_negotiation.cc:407] Connection header received
0812 07:07:35.581663 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:35.581685 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:35.582166 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:35.582610 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:35.584144 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:35.585469 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:35.587007 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:35.587748 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:35.588010 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:35.593239 (+ 5229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:35.593251 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:35.593273 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:35.593317 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:35.597412 (+ 4095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:35.598524 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:35.598532 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:35.598539 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:35.598726 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:35.599218 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:35.599222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:35.599224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:35.599829 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:35.600141 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:35.600677 (+ 536us) server_negotiation.cc:299] Negotiation successful
0812 07:07:35.601167 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":368,"threads_started":1}
I20260812 07:07:35.608055 24713 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf960fc0f93f47f4bbb12e9580b94392" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39783 } }
I20260812 07:07:35.608520 24713 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:07:35.608660 24713 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf960fc0f93f47f4bbb12e9580b94392, State: Initialized, Role: FOLLOWER
I20260812 07:07:35.609347 24713 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf960fc0f93f47f4bbb12e9580b94392" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39783 } }
I20260812 07:07:35.609571 24713 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:07:35.609673 24713 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:07:35.609802 24713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:07:35.612124 24713 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf960fc0f93f47f4bbb12e9580b94392" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39783 } }
I20260812 07:07:35.612624 24713 leader_election.cc:302] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf960fc0f93f47f4bbb12e9580b94392; no voters:
I20260812 07:07:35.613219 24713 leader_election.cc:288] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:07:35.613391 24718 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:07:35.614171 24718 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [term 1 LEADER]: Becoming Leader. State: Replica: cf960fc0f93f47f4bbb12e9580b94392, State: Running, Role: LEADER
I20260812 07:07:35.614831 24718 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf960fc0f93f47f4bbb12e9580b94392" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39783 } }
I20260812 07:07:35.615578 24713 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:07:35.618104 24720 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf960fc0f93f47f4bbb12e9580b94392. Latest consensus state: current_term: 1 leader_uuid: "cf960fc0f93f47f4bbb12e9580b94392" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf960fc0f93f47f4bbb12e9580b94392" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39783 } } }
I20260812 07:07:35.618069 24719 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf960fc0f93f47f4bbb12e9580b94392" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf960fc0f93f47f4bbb12e9580b94392" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39783 } } }
I20260812 07:07:35.618655 24719 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [sys.catalog]: This master's current role is: LEADER
I20260812 07:07:35.620012 24720 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [sys.catalog]: This master's current role is: LEADER
I20260812 07:07:35.620335 24725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:07:35.628509 24725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:07:35.641680 24725 catalog_manager.cc:1347] Generated new cluster ID: 4f0975c0b8584f9f8832dc83df8f4dd9
I20260812 07:07:35.641868 24725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:07:35.658466 24725 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:07:35.659950 24725 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:07:35.672899 24725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392: Generated new TSK 0
I20260812 07:07:35.673983 24725 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
I20260812 07:07:35.774150 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.1:0
--local_ip_for_outbound_sockets=127.6.61.1
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:39783
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:46191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:07:36.162330 24740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:07:36.162607 24740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:07:36.162771 24740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:07:36.170229 24740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:07:36.170343 24740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:07:36.170517 24740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.1
I20260812 07:07:36.180460 24740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:46191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:39783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.1
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:07:36.181787 24740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:07:36.183166 24740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:07:36.196646 24745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:36.196596 24748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:36.196580 24746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:07:36.197463 24740 server_base.cc:1034] running on GCE node
I20260812 07:07:36.198194 24740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:07:36.199164 24740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:07:36.200385 24740 hybrid_clock.cc:648] HybridClock initialized: now 1786518456200340 us; error 32 us; skew 500 ppm
I20260812 07:07:36.200728 24740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:07:36 dist-test-slave-69rg krb5kdc[24621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518456, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:07:36.207706 24740 init.cc:377] Logged in from keytab as kudu/127.6.61.1@KRBTEST.COM (short username kudu)
I20260812 07:07:36.209463 24740 webserver.cc:460] Webserver started at http://127.6.61.1:41937/ using document root <none> and password file <none>
I20260812 07:07:36.210407 24740 fs_manager.cc:359] Metadata directory not provided
I20260812 07:07:36.210572 24740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:07:36.210924 24740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:07:36.214226 24740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/data/instance:
uuid: "b3aad6d4634947aca8c854dad3fb9401"
format_stamp: "Formatted at 2026-08-12 07:07:36 on dist-test-slave-69rg"
server_key: "1726941850eaa84ad8046bd670747d3b"
server_key_iv: "c522ff3207f99a1056bcea945dc9bb0e"
server_key_version: "encryptionkey@0"
I20260812 07:07:36.215042 24740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance:
uuid: "b3aad6d4634947aca8c854dad3fb9401"
format_stamp: "Formatted at 2026-08-12 07:07:36 on dist-test-slave-69rg"
server_key: "1726941850eaa84ad8046bd670747d3b"
server_key_iv: "c522ff3207f99a1056bcea945dc9bb0e"
server_key_version: "encryptionkey@0"
I20260812 07:07:36.220847 24740 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 07:07:36.225016 24756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:36.226608 24740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:07:36.226823 24740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/wal
uuid: "b3aad6d4634947aca8c854dad3fb9401"
format_stamp: "Formatted at 2026-08-12 07:07:36 on dist-test-slave-69rg"
server_key: "1726941850eaa84ad8046bd670747d3b"
server_key_iv: "c522ff3207f99a1056bcea945dc9bb0e"
server_key_version: "encryptionkey@0"
I20260812 07:07:36.227069 24740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:07:36.255743 24740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:07:36.260421 24740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:07:36.260762 24740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:07:36.261951 24740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:07:36.263345 24740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:07:36.263454 24740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:36.263607 24740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:07:36.263672 24740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:36.299717 24740 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.1:35167
I20260812 07:07:36.299822 24888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.1:35167 every 8 connection(s)
I20260812 07:07:36.301050 24740 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
I20260812 07:07:36.302485 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 24740
I20260812 07:07:36.302755 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance
I20260812 07:07:36.303633 6388 external_mini_cluster.cc:1428] Setting key 3d0cbe327ac08260f22e41fc5a5e5711
Aug 12 07:07:36 dist-test-slave-69rg krb5kdc[24621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518456, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:07:36.328236 24891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:36.304224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:58251 (local address 127.6.61.62:39783)
0812 07:07:36.304758 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:36.304763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:36.305870 (+ 1107us) server_negotiation.cc:407] Connection header received
0812 07:07:36.308902 (+ 3032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:36.308906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:36.308989 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:36.309120 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:36.311468 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:36.312256 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:36.314952 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:36.315453 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:36.315574 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:36.320538 (+ 4964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:36.320545 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:36.320548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:36.320587 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:36.324248 (+ 3661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:36.325278 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:36.325287 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:36.325293 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:36.325459 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:36.325999 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:36.326007 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:36.326012 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:36.326430 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:36.326593 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:36.327422 (+ 829us) server_negotiation.cc:299] Negotiation successful
0812 07:07:36.327737 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":265,"threads_started":1}
I20260812 07:07:36.328960 24892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:36.304855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.1:58251)
0812 07:07:36.305678 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:36.305717 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:36.308525 (+ 2808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:36.309427 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:36.309450 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:36.310344 (+ 894us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:07:36.311237 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:36.311268 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:36.312437 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:36.312446 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:36.314747 (+ 2301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:36.314774 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:36.315715 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:36.315726 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:36.316099 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:36.317022 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:07:36.317066 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:07:36.320225 (+ 3159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:07:36.324445 (+ 4220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:36.324453 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:36.324469 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:36.325121 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:36.325597 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:36.325602 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:36.325604 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:36.325802 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:36.326590 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:07:36.326597 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:07:36.326958 (+ 361us) client_negotiation.cc:769] Sending connection context
0812 07:07:36.327226 (+ 268us) client_negotiation.cc:240] Negotiation successful
0812 07:07:36.327672 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":322,"threads_started":1}
I20260812 07:07:36.330890 24889 heartbeater.cc:347] Connected to a master server at 127.6.61.62:39783
I20260812 07:07:36.331303 24889 heartbeater.cc:464] Registering TS with master...
I20260812 07:07:36.332158 24889 heartbeater.cc:511] Master 127.6.61.62:39783 requested a full tablet report, sending...
I20260812 07:07:36.334420 24674 ts_manager.cc:194] Registered new tserver with Master: b3aad6d4634947aca8c854dad3fb9401 (127.6.61.1:35167)
I20260812 07:07:36.335948 24674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.1@KRBTEST.COM'} at 127.6.61.1:58251
WARNING: no policy specified for kudu/127.6.61.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
I20260812 07:07:36.376958 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.2:0
--local_ip_for_outbound_sockets=127.6.61.2
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:39783
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:46191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:07:36.755191 24895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:07:36.755640 24895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:07:36.755811 24895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:07:36.769970 24895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:07:36.770141 24895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:07:36.770381 24895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.2
I20260812 07:07:36.789196 24895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:46191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:39783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.2
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:07:36.791023 24895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:07:36.792918 24895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:07:36.811246 24902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:36.811193 24905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:36.812188 24903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:07:36.812774 24895 server_base.cc:1034] running on GCE node
I20260812 07:07:36.813527 24895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:07:36.814626 24895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:07:36.815872 24895 hybrid_clock.cc:648] HybridClock initialized: now 1786518456815811 us; error 68 us; skew 500 ppm
I20260812 07:07:36.816241 24895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:07:36 dist-test-slave-69rg krb5kdc[24621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518456, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:07:36.819700 24895 init.cc:377] Logged in from keytab as kudu/127.6.61.2@KRBTEST.COM (short username kudu)
I20260812 07:07:36.821228 24895 webserver.cc:460] Webserver started at http://127.6.61.2:46073/ using document root <none> and password file <none>
I20260812 07:07:36.821969 24895 fs_manager.cc:359] Metadata directory not provided
I20260812 07:07:36.822084 24895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:07:36.822355 24895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:07:36.825344 24895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/data/instance:
uuid: "49e9a79960254210b48fe8468c593d01"
format_stamp: "Formatted at 2026-08-12 07:07:36 on dist-test-slave-69rg"
server_key: "e8ebd3b0fcc47f2792330901dea24982"
server_key_iv: "151e547d0e650c7cbc6361fb9a5ef693"
server_key_version: "encryptionkey@0"
I20260812 07:07:36.826134 24895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance:
uuid: "49e9a79960254210b48fe8468c593d01"
format_stamp: "Formatted at 2026-08-12 07:07:36 on dist-test-slave-69rg"
server_key: "e8ebd3b0fcc47f2792330901dea24982"
server_key_iv: "151e547d0e650c7cbc6361fb9a5ef693"
server_key_version: "encryptionkey@0"
I20260812 07:07:36.831657 24895 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 07:07:36.835186 24912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:36.836498 24895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:07:36.836680 24895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/wal
uuid: "49e9a79960254210b48fe8468c593d01"
format_stamp: "Formatted at 2026-08-12 07:07:36 on dist-test-slave-69rg"
server_key: "e8ebd3b0fcc47f2792330901dea24982"
server_key_iv: "151e547d0e650c7cbc6361fb9a5ef693"
server_key_version: "encryptionkey@0"
I20260812 07:07:36.836891 24895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:07:36.861640 24895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:07:36.866137 24895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:07:36.866459 24895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:07:36.867493 24895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:07:36.869009 24895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:07:36.869112 24895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:36.869270 24895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:07:36.869347 24895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:36.906381 24895 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.2:40553
I20260812 07:07:36.906484 25041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.2:40553 every 8 connection(s)
I20260812 07:07:36.907706 24895 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
I20260812 07:07:36.910228 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 24895
I20260812 07:07:36.910452 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance
I20260812 07:07:36.911142 6388 external_mini_cluster.cc:1428] Setting key c2c1f99ad6ee550db819232bf48863a8
Aug 12 07:07:36 dist-test-slave-69rg krb5kdc[24621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518456, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:07:36.928927 25044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:36.910595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:58873 (local address 127.6.61.62:39783)
0812 07:07:36.911116 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:36.911124 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:36.912236 (+ 1112us) server_negotiation.cc:407] Connection header received
0812 07:07:36.914155 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:36.914160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:36.914253 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:36.914377 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:36.915908 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:36.916665 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:36.918413 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:36.918866 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:36.919010 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:36.922568 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:36.922573 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:36.922577 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:36.922610 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:36.925408 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:36.926449 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:36.926453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:36.926455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:36.926584 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:36.927088 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:36.927091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:36.927093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:36.927372 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:36.927488 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:36.928449 (+ 961us) server_negotiation.cc:299] Negotiation successful
0812 07:07:36.928644 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":250,"threads_started":1}
I20260812 07:07:36.930598 25045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:36.911161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.2:58873)
0812 07:07:36.912034 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:36.912089 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:36.913873 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:36.914624 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:36.914633 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:36.915135 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:07:36.915706 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:36.915724 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:36.916824 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:36.916831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:36.918251 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:36.918273 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:36.919101 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:36.919110 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:36.919353 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:36.920067 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:07:36.920089 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:07:36.922331 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:07:36.925563 (+ 3232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:36.925574 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:36.925595 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:36.926316 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:36.926702 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:36.926708 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:36.926714 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:36.926968 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:36.927511 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:07:36.927522 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:07:36.927980 (+ 458us) client_negotiation.cc:769] Sending connection context
0812 07:07:36.928297 (+ 317us) client_negotiation.cc:240] Negotiation successful
0812 07:07:36.928891 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":361,"threads_started":1}
I20260812 07:07:36.932333 25042 heartbeater.cc:347] Connected to a master server at 127.6.61.62:39783
I20260812 07:07:36.932725 25042 heartbeater.cc:464] Registering TS with master...
I20260812 07:07:36.933606 25042 heartbeater.cc:511] Master 127.6.61.62:39783 requested a full tablet report, sending...
I20260812 07:07:36.935397 24674 ts_manager.cc:194] Registered new tserver with Master: 49e9a79960254210b48fe8468c593d01 (127.6.61.2:40553)
I20260812 07:07:36.936717 24674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.2@KRBTEST.COM'} at 127.6.61.2:58873
WARNING: no policy specified for kudu/127.6.61.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
I20260812 07:07:36.985512 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.3:0
--local_ip_for_outbound_sockets=127.6.61.3
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:39783
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:46191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:07:37.337369 25048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:07:37.337721 25048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:07:37.337924 25048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260812 07:07:37.338970 24889 heartbeater.cc:503] Master 127.6.61.62:39783 was elected leader, sending a full tablet report...
W20260812 07:07:37.346287 25048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:07:37.346454 25048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:07:37.346652 25048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.3
I20260812 07:07:37.356945 25048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:46191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:39783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.3
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:07:37.358453 25048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:07:37.359795 25048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:07:37.373612 25053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:37.373672 25054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:37.373581 25057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:07:37.374457 25048 server_base.cc:1034] running on GCE node
I20260812 07:07:37.375131 25048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:07:37.376187 25048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:07:37.377414 25048 hybrid_clock.cc:648] HybridClock initialized: now 1786518457377344 us; error 56 us; skew 500 ppm
I20260812 07:07:37.377779 25048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:07:37 dist-test-slave-69rg krb5kdc[24621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518457, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:07:37.381156 25048 init.cc:377] Logged in from keytab as kudu/127.6.61.3@KRBTEST.COM (short username kudu)
I20260812 07:07:37.382831 25048 webserver.cc:460] Webserver started at http://127.6.61.3:36949/ using document root <none> and password file <none>
I20260812 07:07:37.383541 25048 fs_manager.cc:359] Metadata directory not provided
I20260812 07:07:37.383702 25048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:07:37.384038 25048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:07:37.387255 25048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/data/instance:
uuid: "9639a9db44224890a8fcaaf1486080f5"
format_stamp: "Formatted at 2026-08-12 07:07:37 on dist-test-slave-69rg"
server_key: "bb48cd9d98f1e2087020ec6f5eac142c"
server_key_iv: "5653395901af7eb1d2e2234c552aed9e"
server_key_version: "encryptionkey@0"
I20260812 07:07:37.388063 25048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance:
uuid: "9639a9db44224890a8fcaaf1486080f5"
format_stamp: "Formatted at 2026-08-12 07:07:37 on dist-test-slave-69rg"
server_key: "bb48cd9d98f1e2087020ec6f5eac142c"
server_key_iv: "5653395901af7eb1d2e2234c552aed9e"
server_key_version: "encryptionkey@0"
I20260812 07:07:37.393731 25048 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 07:07:37.397543 25064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:37.399020 25048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260812 07:07:37.399224 25048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/wal
uuid: "9639a9db44224890a8fcaaf1486080f5"
format_stamp: "Formatted at 2026-08-12 07:07:37 on dist-test-slave-69rg"
server_key: "bb48cd9d98f1e2087020ec6f5eac142c"
server_key_iv: "5653395901af7eb1d2e2234c552aed9e"
server_key_version: "encryptionkey@0"
I20260812 07:07:37.399456 25048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:07:37.416769 25048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:07:37.421109 25048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:07:37.421438 25048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:07:37.422575 25048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:07:37.424062 25048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:07:37.424271 25048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260812 07:07:37.424435 25048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:07:37.424536 25048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:37.460229 25048 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.3:46821
I20260812 07:07:37.460335 25197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.3:46821 every 8 connection(s)
I20260812 07:07:37.461567 25048 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
I20260812 07:07:37.467353 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 25048
I20260812 07:07:37.467562 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance
I20260812 07:07:37.468117 6388 external_mini_cluster.cc:1428] Setting key 9162e7b7b2dbc8225a0ac64574863e06
Aug 12 07:07:37 dist-test-slave-69rg krb5kdc[24621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518457, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:07:37.483628 25200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:37.464585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:42779 (local address 127.6.61.62:39783)
0812 07:07:37.465086 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:37.465091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:37.466126 (+ 1035us) server_negotiation.cc:407] Connection header received
0812 07:07:37.468148 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:37.468153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:37.468252 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:37.468386 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:37.470346 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:37.471584 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:37.473172 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:37.473560 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:37.473657 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:37.477396 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:37.477402 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:37.477405 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:37.477439 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:37.480295 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:37.481055 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:37.481059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:37.481062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:37.481182 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:37.481651 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:37.481655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:37.481657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:37.481994 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:37.482113 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:37.483053 (+ 940us) server_negotiation.cc:299] Negotiation successful
0812 07:07:37.483238 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":262,"threads_started":1}
I20260812 07:07:37.484851 25201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:37.465135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.3:42779)
0812 07:07:37.465961 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:37.466003 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:37.467865 (+ 1862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:37.468633 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:37.468642 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:37.469313 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:07:37.470084 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:37.470129 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:37.471732 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:37.471736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:37.473040 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:37.473053 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:37.473794 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:37.473800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:37.474087 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:37.474757 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:07:37.474781 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:07:37.477171 (+ 2390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:07:37.480443 (+ 3272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:37.480451 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:37.480465 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:37.480938 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:37.481312 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:37.481316 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:37.481319 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:37.481538 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:37.482139 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:07:37.482154 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:07:37.482572 (+ 418us) client_negotiation.cc:769] Sending connection context
0812 07:07:37.482910 (+ 338us) client_negotiation.cc:240] Negotiation successful
0812 07:07:37.483501 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":302,"threads_started":1}
I20260812 07:07:37.486447 25198 heartbeater.cc:347] Connected to a master server at 127.6.61.62:39783
I20260812 07:07:37.486810 25198 heartbeater.cc:464] Registering TS with master...
I20260812 07:07:37.487773 25198 heartbeater.cc:511] Master 127.6.61.62:39783 requested a full tablet report, sending...
I20260812 07:07:37.489429 24674 ts_manager.cc:194] Registered new tserver with Master: 9639a9db44224890a8fcaaf1486080f5 (127.6.61.3:46821)
I20260812 07:07:37.490795 24674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.3@KRBTEST.COM'} at 127.6.61.3:42779
I20260812 07:07:37.497318 6388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 07:07:37.516808 25200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:37.503570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36372 (local address 127.6.61.62:39783)
0812 07:07:37.503751 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:37.503755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:37.503790 (+ 35us) server_negotiation.cc:407] Connection header received
0812 07:07:37.503851 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:37.503859 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:37.503949 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:37.504057 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:37.505904 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:37.506664 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:37.508138 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:37.508574 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:37.508672 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:37.511409 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:37.511414 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:37.511417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:37.511448 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:37.514102 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:37.514839 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:37.514844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:37.514846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:37.514965 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:37.515384 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:37.515389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:37.515392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:37.515828 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:37.516013 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:37.516327 (+ 314us) server_negotiation.cc:299] Negotiation successful
0812 07:07:37.516501 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 07:07:37.521236 24674 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:07:37.528457 24674 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:07:37.568282 25219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:37.542498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54384 (local address 127.6.61.2:40553)
0812 07:07:37.554825 (+ 12327us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:37.554839 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:37.560306 (+ 5467us) server_negotiation.cc:407] Connection header received
0812 07:07:37.560421 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:37.560428 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:37.560661 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:37.560876 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:07:37.561644 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:37.563261 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:37.565361 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:37.566705 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:37.566856 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:37.567091 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:37.567722 (+ 631us) server_negotiation.cc:299] Negotiation successful
0812 07:07:37.567917 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12106,"thread_start_us":12341,"threads_started":1}
I20260812 07:07:37.568334 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:37.542234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:40553 (local address 127.0.0.1:54384)
0812 07:07:37.559691 (+ 17457us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:37.559710 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:37.559834 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:37.560906 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:37.560910 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:37.560937 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:07:37.561433 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:37.561444 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:37.563438 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:37.563445 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:37.565182 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:37.565204 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:37.567244 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:37.567250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:37.567444 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:37.567495 (+ 51us) client_negotiation.cc:769] Sending connection context
0812 07:07:37.567644 (+ 149us) client_negotiation.cc:240] Negotiation successful
0812 07:07:37.567842 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17310,"mutex_wait_us":1,"thread_start_us":17136,"threads_started":1}
I20260812 07:07:37.569247 25212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:37.541304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:46821 (local address 127.0.0.1:37228)
0812 07:07:37.552668 (+ 11364us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:37.552766 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:37.554498 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:37.561195 (+ 6697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:37.561200 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:37.561228 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:07:37.561730 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:37.561744 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:37.563649 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:37.563656 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:37.566144 (+ 2488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:37.566158 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:37.568182 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:37.568189 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:37.568390 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:37.568428 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 07:07:37.568511 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 07:07:37.568615 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10920,"mutex_wait_us":48,"thread_start_us":10707,"threads_started":1}
I20260812 07:07:37.569291 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:37.541605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37228 (local address 127.6.61.3:46821)
0812 07:07:37.560448 (+ 18843us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:37.560455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:37.560480 (+ 25us) server_negotiation.cc:407] Connection header received
0812 07:07:37.560556 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:37.560560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:37.560721 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:37.560933 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:07:37.561952 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:37.563512 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:37.566358 (+ 2846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:37.567678 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:37.567805 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:37.568022 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:37.568724 (+ 702us) server_negotiation.cc:299] Negotiation successful
0812 07:07:37.568925 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18666,"thread_start_us":18493,"threads_started":1}
I20260812 07:07:37.570073 25218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:37.541776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:35167 (local address 127.0.0.1:42300)
0812 07:07:37.558457 (+ 16681us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:37.558485 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:37.558674 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:37.560433 (+ 1759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:37.560441 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:37.560473 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:07:37.560931 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:37.560950 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:37.562957 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:37.562962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:37.565772 (+ 2810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:37.565789 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:37.568937 (+ 3148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:37.568944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:37.569123 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:37.569158 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 07:07:37.569259 (+ 101us) client_negotiation.cc:240] Negotiation successful
0812 07:07:37.569664 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16503,"mutex_wait_us":19,"thread_start_us":16325,"threads_started":1}
I20260812 07:07:37.573318 24969 tablet_service.cc:1467] Processing CreateTablet for tablet aa8a0d50516549e8be78a2def26b80d4 (DEFAULT_TABLE table=test-table [id=9e2f70b2c18d4c3aad60ed86c99598d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:07:37.573319 25119 tablet_service.cc:1467] Processing CreateTablet for tablet aa8a0d50516549e8be78a2def26b80d4 (DEFAULT_TABLE table=test-table [id=9e2f70b2c18d4c3aad60ed86c99598d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:07:37.573954 25215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:37.542220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42300 (local address 127.6.61.1:35167)
0812 07:07:37.559374 (+ 17154us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:37.559385 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:37.559424 (+ 39us) server_negotiation.cc:407] Connection header received
0812 07:07:37.559558 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:37.559568 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:37.559859 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:37.560096 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:07:37.561165 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:37.562787 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:37.565949 (+ 3162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:37.568414 (+ 2465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:37.568536 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:37.568768 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:37.570314 (+ 1546us) server_negotiation.cc:299] Negotiation successful
0812 07:07:37.573618 (+ 3304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16960,"thread_start_us":16772,"threads_started":1}
I20260812 07:07:37.574924 24969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa8a0d50516549e8be78a2def26b80d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:07:37.575727 25119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa8a0d50516549e8be78a2def26b80d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:07:37.577868 24813 tablet_service.cc:1467] Processing CreateTablet for tablet aa8a0d50516549e8be78a2def26b80d4 (DEFAULT_TABLE table=test-table [id=9e2f70b2c18d4c3aad60ed86c99598d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:07:37.579276 24813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa8a0d50516549e8be78a2def26b80d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:07:37.594889 25222 tablet_bootstrap.cc:492] T aa8a0d50516549e8be78a2def26b80d4 P b3aad6d4634947aca8c854dad3fb9401: Bootstrap starting.
I20260812 07:07:37.596685 25220 tablet_bootstrap.cc:492] T aa8a0d50516549e8be78a2def26b80d4 P 49e9a79960254210b48fe8468c593d01: Bootstrap starting.
I20260812 07:07:37.598014 25221 tablet_bootstrap.cc:492] T aa8a0d50516549e8be78a2def26b80d4 P 9639a9db44224890a8fcaaf1486080f5: Bootstrap starting.
I20260812 07:07:37.599467 25222 tablet_bootstrap.cc:654] T aa8a0d50516549e8be78a2def26b80d4 P b3aad6d4634947aca8c854dad3fb9401: Neither blocks nor log segments found. Creating new log.
I20260812 07:07:37.600868 25220 tablet_bootstrap.cc:654] T aa8a0d50516549e8be78a2def26b80d4 P 49e9a79960254210b48fe8468c593d01: Neither blocks nor log segments found. Creating new log.
I20260812 07:07:37.600883 25222 log.cc:824] T aa8a0d50516549e8be78a2def26b80d4 P b3aad6d4634947aca8c854dad3fb9401: Log is configured to *not* fsync() on all Append() calls
I20260812 07:07:37.601577 25221 tablet_bootstrap.cc:654] T aa8a0d50516549e8be78a2def26b80d4 P 9639a9db44224890a8fcaaf1486080f5: Neither blocks nor log segments found. Creating new log.
I20260812 07:07:37.602258 25220 log.cc:824] T aa8a0d50516549e8be78a2def26b80d4 P 49e9a79960254210b48fe8468c593d01: Log is configured to *not* fsync() on all Append() calls
I20260812 07:07:37.602896 25221 log.cc:824] T aa8a0d50516549e8be78a2def26b80d4 P 9639a9db44224890a8fcaaf1486080f5: Log is configured to *not* fsync() on all Append() calls
I20260812 07:07:37.605760 25222 tablet_bootstrap.cc:492] T aa8a0d50516549e8be78a2def26b80d4 P b3aad6d4634947aca8c854dad3fb9401: No bootstrap required, opened a new log
I20260812 07:07:37.605963 25220 tablet_bootstrap.cc:492] T aa8a0d50516549e8be78a2def26b80d4 P 49e9a79960254210b48fe8468c593d01: No bootstrap required, opened a new log
I20260812 07:07:37.606122 25221 tablet_bootstrap.cc:492] T aa8a0d50516549e8be78a2def26b80d4 P 9639a9db44224890a8fcaaf1486080f5: No bootstrap required, opened a new log
I20260812 07:07:37.606154 25222 ts_tablet_manager.cc:1397] T aa8a0d50516549e8be78a2def26b80d4 P b3aad6d4634947aca8c854dad3fb9401: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260812 07:07:37.606247 25220 ts_tablet_manager.cc:1397] T aa8a0d50516549e8be78a2def26b80d4 P 49e9a79960254210b48fe8468c593d01: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260812 07:07:37.606390 25221 ts_tablet_manager.cc:1397] T aa8a0d50516549e8be78a2def26b80d4 P 9639a9db44224890a8fcaaf1486080f5: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260812 07:07:37.614176 25222 raft_consensus.cc:348] T aa8a0d50516549e8be78a2def26b80d4 P b3aad6d4634947aca8c854dad3fb9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9639a9db44224890a8fcaaf1486080f5" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 46821 } } peers { permanent_uuid: "b3aad6d4634947aca8c854dad3fb9401" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 35167 } } peers { permanent_uuid: "49e9a79960254210b48fe8468c593d01" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 40553 } }
I20260812 07:07:37.614621 25222 raft_consensus.cc:374] T aa8a0d50516549e8be78a2def26b80d4 P b3aad6d4634947aca8c854dad3fb9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:07:37.614727 25222 raft_consensus.cc:729] T aa8a0d50516549e8be78a2def26b80d4 P b3aad6d4634947aca8c854dad3fb9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3aad6d4634947aca8c854dad3fb9401, State: Initialized, Role: FOLLOWER
I20260812 07:07:37.615397 25222 consensus_queue.cc:260] T aa8a0d50516549e8be78a2def26b80d4 P b3aad6d4634947aca8c854dad3fb9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9639a9db44224890a8fcaaf1486080f5" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 46821 } } peers { permanent_uuid: "b3aad6d4634947aca8c854dad3fb9401" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 35167 } } peers { permanent_uuid: "49e9a79960254210b48fe8468c593d01" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 40553 } }
I20260812 07:07:37.617377 25222 ts_tablet_manager.cc:1428] T aa8a0d50516549e8be78a2def26b80d4 P b3aad6d4634947aca8c854dad3fb9401: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260812 07:07:37.617513 25220 raft_consensus.cc:348] T aa8a0d50516549e8be78a2def26b80d4 P 49e9a79960254210b48fe8468c593d01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9639a9db44224890a8fcaaf1486080f5" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 46821 } } peers { permanent_uuid: "b3aad6d4634947aca8c854dad3fb9401" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 35167 } } peers { permanent_uuid: "49e9a79960254210b48fe8468c593d01" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 40553 } }
I20260812 07:07:37.617565 25221 raft_consensus.cc:348] T aa8a0d50516549e8be78a2def26b80d4 P 9639a9db44224890a8fcaaf1486080f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9639a9db44224890a8fcaaf1486080f5" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 46821 } } peers { permanent_uuid: "b3aad6d4634947aca8c854dad3fb9401" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 35167 } } peers { permanent_uuid: "49e9a79960254210b48fe8468c593d01" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 40553 } }
I20260812 07:07:37.618062 25220 raft_consensus.cc:374] T aa8a0d50516549e8be78a2def26b80d4 P 49e9a79960254210b48fe8468c593d01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:07:37.618060 25221 raft_consensus.cc:374] T aa8a0d50516549e8be78a2def26b80d4 P 9639a9db44224890a8fcaaf1486080f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:07:37.618206 25221 raft_consensus.cc:729] T aa8a0d50516549e8be78a2def26b80d4 P 9639a9db44224890a8fcaaf1486080f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9639a9db44224890a8fcaaf1486080f5, State: Initialized, Role: FOLLOWER
I20260812 07:07:37.618206 25220 raft_consensus.cc:729] T aa8a0d50516549e8be78a2def26b80d4 P 49e9a79960254210b48fe8468c593d01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e9a79960254210b48fe8468c593d01, State: Initialized, Role: FOLLOWER
I20260812 07:07:37.618881 25220 consensus_queue.cc:260] T aa8a0d50516549e8be78a2def26b80d4 P 49e9a79960254210b48fe8468c593d01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9639a9db44224890a8fcaaf1486080f5" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 46821 } } peers { permanent_uuid: "b3aad6d4634947aca8c854dad3fb9401" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 35167 } } peers { permanent_uuid: "49e9a79960254210b48fe8468c593d01" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 40553 } }
I20260812 07:07:37.618880 25221 consensus_queue.cc:260] T aa8a0d50516549e8be78a2def26b80d4 P 9639a9db44224890a8fcaaf1486080f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9639a9db44224890a8fcaaf1486080f5" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 46821 } } peers { permanent_uuid: "b3aad6d4634947aca8c854dad3fb9401" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 35167 } } peers { permanent_uuid: "49e9a79960254210b48fe8468c593d01" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 40553 } }
I20260812 07:07:37.620157 25042 heartbeater.cc:503] Master 127.6.61.62:39783 was elected leader, sending a full tablet report...
I20260812 07:07:37.620893 25220 ts_tablet_manager.cc:1428] T aa8a0d50516549e8be78a2def26b80d4 P 49e9a79960254210b48fe8468c593d01: Time spent starting tablet: real 0.015s user 0.007s sys 0.007s
I20260812 07:07:37.621024 25198 heartbeater.cc:503] Master 127.6.61.62:39783 was elected leader, sending a full tablet report...
I20260812 07:07:37.622375 25221 ts_tablet_manager.cc:1428] T aa8a0d50516549e8be78a2def26b80d4 P 9639a9db44224890a8fcaaf1486080f5: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
W20260812 07:07:37.659942 25043 tablet.cc:2369] T aa8a0d50516549e8be78a2def26b80d4 P 49e9a79960254210b48fe8468c593d01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:07:37.681783 25227 raft_consensus.cc:482] T aa8a0d50516549e8be78a2def26b80d4 P b3aad6d4634947aca8c854dad3fb9401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:07:37.682183 25227 raft_consensus.cc:504] T aa8a0d50516549e8be78a2def26b80d4 P b3aad6d4634947aca8c854dad3fb9401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9639a9db44224890a8fcaaf1486080f5" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 46821 } } peers { permanent_uuid: "b3aad6d4634947aca8c854dad3fb9401" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 35167 } } peers { permanent_uuid: "49e9a79960254210b48fe8468c593d01" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 40553 } }
I20260812 07:07:37.684194 25227 leader_election.cc:288] T aa8a0d50516549e8be78a2def26b80d4 P b3aad6d4634947aca8c854dad3fb9401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9639a9db44224890a8fcaaf1486080f5 (127.6.61.3:46821), 49e9a79960254210b48fe8468c593d01 (127.6.61.2:40553)
I20260812 07:07:37.694429 25219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:37.684787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:42277 (local address 127.6.61.2:40553)
0812 07:07:37.685044 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:37.685052 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:37.686293 (+ 1241us) server_negotiation.cc:407] Connection header received
0812 07:07:37.686515 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:37.686520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:37.686604 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:37.686675 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:07:37.687497 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:37.688679 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:37.691849 (+ 3170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:37.693125 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:37.693336 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:37.693431 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:37.693997 (+ 566us) server_negotiation.cc:299] Negotiation successful
0812 07:07:37.694141 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 07:07:37.694408 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:37.684754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:36579 (local address 127.6.61.3:46821)
0812 07:07:37.685191 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:37.685197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:37.685897 (+ 700us) server_negotiation.cc:407] Connection header received
0812 07:07:37.686256 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:37.686262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:37.686362 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:37.686466 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:07:37.687231 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:37.688527 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:37.691853 (+ 3326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:37.693134 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:37.693285 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:37.693399 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:37.693997 (+ 598us) server_negotiation.cc:299] Negotiation successful
0812 07:07:37.694141 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273}
I20260812 07:07:37.694638 25231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:37.684790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:40553 (local address 127.6.61.1:42277)
0812 07:07:37.686169 (+ 1379us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:37.686186 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:37.686336 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:37.686847 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:37.686850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:37.686874 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:07:37.687311 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:37.687319 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:37.688844 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:37.688856 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:37.691630 (+ 2774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:37.691648 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:37.693472 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:37.693479 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:37.693734 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:37.693847 (+ 113us) client_negotiation.cc:769] Sending connection context
0812 07:07:37.693949 (+ 102us) client_negotiation.cc:240] Negotiation successful
0812 07:07:37.694128 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1233,"mutex_wait_us":315,"thread_start_us":542,"threads_started":1}
I20260812 07:07:37.694692 25233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:37.684642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:46821 (local address 127.6.61.1:36579)
0812 07:07:37.685736 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:37.685767 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:37.686051 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:37.686513 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:37.686519 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:37.686551 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:07:37.687037 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:37.687051 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:37.688708 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:37.688712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:37.691634 (+ 2922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:37.691655 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:37.693489 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:37.693495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:37.693734 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:37.693845 (+ 111us) client_negotiation.cc:769] Sending connection context
0812 07:07:37.693949 (+ 104us) client_negotiation.cc:240] Negotiation successful
0812 07:07:37.694128 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"mutex_wait_us":411,"thread_start_us":1127,"threads_started":1}
I20260812 07:07:37.695771 25144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8a0d50516549e8be78a2def26b80d4" candidate_uuid: "b3aad6d4634947aca8c854dad3fb9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9639a9db44224890a8fcaaf1486080f5" is_pre_election: true
I20260812 07:07:37.695820 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8a0d50516549e8be78a2def26b80d4" candidate_uuid: "b3aad6d4634947aca8c854dad3fb9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e9a79960254210b48fe8468c593d01" is_pre_election: true
I20260812 07:07:37.696364 24990 raft_consensus.cc:2454] T aa8a0d50516549e8be78a2def26b80d4 P 49e9a79960254210b48fe8468c593d01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3aad6d4634947aca8c854dad3fb9401 in term 0.
I20260812 07:07:37.696363 25144 raft_consensus.cc:2454] T aa8a0d50516549e8be78a2def26b80d4 P 9639a9db44224890a8fcaaf1486080f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3aad6d4634947aca8c854dad3fb9401 in term 0.
I20260812 07:07:37.697059 24758 leader_election.cc:302] T aa8a0d50516549e8be78a2def26b80d4 P b3aad6d4634947aca8c854dad3fb9401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e9a79960254210b48fe8468c593d01, b3aad6d4634947aca8c854dad3fb9401; no voters:
I20260812 07:07:37.697505 25227 raft_consensus.cc:2781] T aa8a0d50516549e8be78a2def26b80d4 P b3aad6d4634947aca8c854dad3fb9401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:07:37.697659 25227 raft_consensus.cc:482] T aa8a0d50516549e8be78a2def26b80d4 P b3aad6d4634947aca8c854dad3fb9401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:07:37.697851 25227 raft_consensus.cc:3037] T aa8a0d50516549e8be78a2def26b80d4 P b3aad6d4634947aca8c854dad3fb9401 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:07:37.700543 25227 raft_consensus.cc:504] T aa8a0d50516549e8be78a2def26b80d4 P b3aad6d4634947aca8c854dad3fb9401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9639a9db44224890a8fcaaf1486080f5" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 46821 } } peers { permanent_uuid: "b3aad6d4634947aca8c854dad3fb9401" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 35167 } } peers { permanent_uuid: "49e9a79960254210b48fe8468c593d01" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 40553 } }
I20260812 07:07:37.701246 25227 leader_election.cc:288] T aa8a0d50516549e8be78a2def26b80d4 P b3aad6d4634947aca8c854dad3fb9401 [CANDIDATE]: Term 1 election: Requested vote from peers 9639a9db44224890a8fcaaf1486080f5 (127.6.61.3:46821), 49e9a79960254210b48fe8468c593d01 (127.6.61.2:40553)
I20260812 07:07:37.701838 25144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8a0d50516549e8be78a2def26b80d4" candidate_uuid: "b3aad6d4634947aca8c854dad3fb9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9639a9db44224890a8fcaaf1486080f5"
I20260812 07:07:37.701943 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8a0d50516549e8be78a2def26b80d4" candidate_uuid: "b3aad6d4634947aca8c854dad3fb9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e9a79960254210b48fe8468c593d01"
I20260812 07:07:37.702124 25144 raft_consensus.cc:3037] T aa8a0d50516549e8be78a2def26b80d4 P 9639a9db44224890a8fcaaf1486080f5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:07:37.702173 24990 raft_consensus.cc:3037] T aa8a0d50516549e8be78a2def26b80d4 P 49e9a79960254210b48fe8468c593d01 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:07:37.705524 24990 raft_consensus.cc:2454] T aa8a0d50516549e8be78a2def26b80d4 P 49e9a79960254210b48fe8468c593d01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3aad6d4634947aca8c854dad3fb9401 in term 1.
I20260812 07:07:37.705524 25144 raft_consensus.cc:2454] T aa8a0d50516549e8be78a2def26b80d4 P 9639a9db44224890a8fcaaf1486080f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3aad6d4634947aca8c854dad3fb9401 in term 1.
I20260812 07:07:37.706003 24758 leader_election.cc:302] T aa8a0d50516549e8be78a2def26b80d4 P b3aad6d4634947aca8c854dad3fb9401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e9a79960254210b48fe8468c593d01, b3aad6d4634947aca8c854dad3fb9401; no voters:
I20260812 07:07:37.706382 25227 raft_consensus.cc:2781] T aa8a0d50516549e8be78a2def26b80d4 P b3aad6d4634947aca8c854dad3fb9401 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:07:37.706892 25227 raft_consensus.cc:686] T aa8a0d50516549e8be78a2def26b80d4 P b3aad6d4634947aca8c854dad3fb9401 [term 1 LEADER]: Becoming Leader. State: Replica: b3aad6d4634947aca8c854dad3fb9401, State: Running, Role: LEADER
I20260812 07:07:37.707453 25227 consensus_queue.cc:237] T aa8a0d50516549e8be78a2def26b80d4 P b3aad6d4634947aca8c854dad3fb9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9639a9db44224890a8fcaaf1486080f5" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 46821 } } peers { permanent_uuid: "b3aad6d4634947aca8c854dad3fb9401" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 35167 } } peers { permanent_uuid: "49e9a79960254210b48fe8468c593d01" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 40553 } }
I20260812 07:07:37.712283 24672 catalog_manager.cc:5495] T aa8a0d50516549e8be78a2def26b80d4 P b3aad6d4634947aca8c854dad3fb9401 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3aad6d4634947aca8c854dad3fb9401 (127.6.61.1). New cstate: current_term: 1 leader_uuid: "b3aad6d4634947aca8c854dad3fb9401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9639a9db44224890a8fcaaf1486080f5" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 46821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3aad6d4634947aca8c854dad3fb9401" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 35167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49e9a79960254210b48fe8468c593d01" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 40553 } health_report { overall_health: UNKNOWN } } }
W20260812 07:07:37.713379 25199 tablet.cc:2369] T aa8a0d50516549e8be78a2def26b80d4 P 9639a9db44224890a8fcaaf1486080f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:07:37.729898 25215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:37.723431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42302 (local address 127.6.61.1:35167)
0812 07:07:37.723685 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:37.723695 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:37.723718 (+ 23us) server_negotiation.cc:407] Connection header received
0812 07:07:37.724019 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:37.724024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:37.724196 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:37.724395 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:07:37.724941 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:37.725734 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:37.727344 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:37.727821 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:37.727977 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:37.728587 (+ 610us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:07:37.729171 (+ 584us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:07:37.729345 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:37.729436 (+ 91us) server_negotiation.cc:299] Negotiation successful
0812 07:07:37.729523 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260812 07:07:37.739043 25144 raft_consensus.cc:1261] T aa8a0d50516549e8be78a2def26b80d4 P 9639a9db44224890a8fcaaf1486080f5 [term 1 FOLLOWER]: Refusing update from remote peer b3aad6d4634947aca8c854dad3fb9401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:07:37.739043 24990 raft_consensus.cc:1261] T aa8a0d50516549e8be78a2def26b80d4 P 49e9a79960254210b48fe8468c593d01 [term 1 FOLLOWER]: Refusing update from remote peer b3aad6d4634947aca8c854dad3fb9401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:07:37.740218 25234 consensus_queue.cc:1035] T aa8a0d50516549e8be78a2def26b80d4 P b3aad6d4634947aca8c854dad3fb9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e9a79960254210b48fe8468c593d01" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:07:37.740619 25227 consensus_queue.cc:1035] T aa8a0d50516549e8be78a2def26b80d4 P b3aad6d4634947aca8c854dad3fb9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9639a9db44224890a8fcaaf1486080f5" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:07:37.761965 25240 mvcc.cc:205] Tried to move back new op lower bound from 7317579602886799360 to 7317579602773688320. Current Snapshot: MvccSnapshot[applied={T|T < 7317579602886799360 or (T in {7317579602886799360})}]
I20260812 07:07:37.771359 24673 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:07:37.771664 24673 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:07:37.775223 24673 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392: Sending DeleteTablet for 3 replicas of tablet aa8a0d50516549e8be78a2def26b80d4
I20260812 07:07:37.776505 24813 tablet_service.cc:1514] Processing DeleteTablet for tablet aa8a0d50516549e8be78a2def26b80d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:07:37 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:42300
I20260812 07:07:37.776504 25119 tablet_service.cc:1514] Processing DeleteTablet for tablet aa8a0d50516549e8be78a2def26b80d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:07:37 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:37228
I20260812 07:07:37.776585 24969 tablet_service.cc:1514] Processing DeleteTablet for tablet aa8a0d50516549e8be78a2def26b80d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:07:37 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:54384
I20260812 07:07:37.778988 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 24740
I20260812 07:07:37.789276 25253 tablet_replica.cc:330] T aa8a0d50516549e8be78a2def26b80d4 P 49e9a79960254210b48fe8468c593d01: stopping tablet replica
I20260812 07:07:37.789840 25253 raft_consensus.cc:2229] T aa8a0d50516549e8be78a2def26b80d4 P 49e9a79960254210b48fe8468c593d01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:07:37.790405 25253 raft_consensus.cc:2258] T aa8a0d50516549e8be78a2def26b80d4 P 49e9a79960254210b48fe8468c593d01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:07:37.792775 25253 ts_tablet_manager.cc:1905] T aa8a0d50516549e8be78a2def26b80d4 P 49e9a79960254210b48fe8468c593d01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:07:37.793993 25248 tablet_replica.cc:330] T aa8a0d50516549e8be78a2def26b80d4 P 9639a9db44224890a8fcaaf1486080f5: stopping tablet replica
I20260812 07:07:37.794440 25248 raft_consensus.cc:2229] T aa8a0d50516549e8be78a2def26b80d4 P 9639a9db44224890a8fcaaf1486080f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:07:37.794927 25248 raft_consensus.cc:2258] T aa8a0d50516549e8be78a2def26b80d4 P 9639a9db44224890a8fcaaf1486080f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:07:37.795722 25252 tablet_replica.cc:330] T aa8a0d50516549e8be78a2def26b80d4 P b3aad6d4634947aca8c854dad3fb9401: stopping tablet replica
I20260812 07:07:37.796186 25252 raft_consensus.cc:2229] T aa8a0d50516549e8be78a2def26b80d4 P b3aad6d4634947aca8c854dad3fb9401 [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:07:37.796862 25252 raft_consensus.cc:2258] T aa8a0d50516549e8be78a2def26b80d4 P b3aad6d4634947aca8c854dad3fb9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:07:37.800910 25252 ts_tablet_manager.cc:1905] T aa8a0d50516549e8be78a2def26b80d4 P b3aad6d4634947aca8c854dad3fb9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:07:37.800944 25253 ts_tablet_manager.cc:1918] T aa8a0d50516549e8be78a2def26b80d4 P 49e9a79960254210b48fe8468c593d01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:07:37.801126 25253 log.cc:1196] T aa8a0d50516549e8be78a2def26b80d4 P 49e9a79960254210b48fe8468c593d01: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/wal/wals/aa8a0d50516549e8be78a2def26b80d4
I20260812 07:07:37.801692 25253 ts_tablet_manager.cc:1939] T aa8a0d50516549e8be78a2def26b80d4 P 49e9a79960254210b48fe8468c593d01: Deleting consensus metadata
I20260812 07:07:37.802547 25248 ts_tablet_manager.cc:1905] T aa8a0d50516549e8be78a2def26b80d4 P 9639a9db44224890a8fcaaf1486080f5: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 12 07:07:37 dist-test-slave-69rg krb5kdc[24621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.1@KRBTEST.COM
I20260812 07:07:37.803774 24657 catalog_manager.cc:4841] TS 49e9a79960254210b48fe8468c593d01 (127.6.61.2:40553): tablet aa8a0d50516549e8be78a2def26b80d4 (table test-table [id=9e2f70b2c18d4c3aad60ed86c99598d3]) successfully deleted
I20260812 07:07:37.810071 25248 ts_tablet_manager.cc:1918] T aa8a0d50516549e8be78a2def26b80d4 P 9639a9db44224890a8fcaaf1486080f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:07:37.810261 25248 log.cc:1196] T aa8a0d50516549e8be78a2def26b80d4 P 9639a9db44224890a8fcaaf1486080f5: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/wal/wals/aa8a0d50516549e8be78a2def26b80d4
I20260812 07:07:37.810842 25248 ts_tablet_manager.cc:1939] T aa8a0d50516549e8be78a2def26b80d4 P 9639a9db44224890a8fcaaf1486080f5: Deleting consensus metadata
I20260812 07:07:37.811278 25252 ts_tablet_manager.cc:1918] T aa8a0d50516549e8be78a2def26b80d4 P b3aad6d4634947aca8c854dad3fb9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:07:37.811466 25252 log.cc:1196] T aa8a0d50516549e8be78a2def26b80d4 P b3aad6d4634947aca8c854dad3fb9401: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/wal/wals/aa8a0d50516549e8be78a2def26b80d4
I20260812 07:07:37.812008 25252 ts_tablet_manager.cc:1939] T aa8a0d50516549e8be78a2def26b80d4 P b3aad6d4634947aca8c854dad3fb9401: Deleting consensus metadata
I20260812 07:07:37.812875 24659 catalog_manager.cc:4841] TS 9639a9db44224890a8fcaaf1486080f5 (127.6.61.3:46821): tablet aa8a0d50516549e8be78a2def26b80d4 (table test-table [id=9e2f70b2c18d4c3aad60ed86c99598d3]) successfully deleted
I20260812 07:07:37.813833 24656 catalog_manager.cc:4841] TS b3aad6d4634947aca8c854dad3fb9401 (127.6.61.1:35167): tablet aa8a0d50516549e8be78a2def26b80d4 (table test-table [id=9e2f70b2c18d4c3aad60ed86c99598d3]) successfully deleted
I20260812 07:07:37.814713 25215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:37.780390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42306 (local address 127.6.61.1:35167)
0812 07:07:37.780619 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:37.780629 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:37.786100 (+ 5471us) server_negotiation.cc:407] Connection header received
0812 07:07:37.787307 (+ 1207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:37.787316 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:37.787430 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:37.787541 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:37.793805 (+ 6264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:37.795026 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:37.797532 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:37.798323 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:37.798428 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:37.805534 (+ 7106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:37.805549 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:37.805568 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:37.805621 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:37.810863 (+ 5242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:37.812157 (+ 1294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:37.812168 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:37.812171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:37.812376 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:37.812926 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:37.812931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:37.812934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:37.813467 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:37.813795 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:37.814251 (+ 456us) server_negotiation.cc:299] Negotiation successful
0812 07:07:37.814440 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 07:07:37.815565 24883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:07:37.918604 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 24740
I20260812 07:07:37.946388 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 24895
Aug 12 07:07:37 dist-test-slave-69rg krb5kdc[24621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.2@KRBTEST.COM
I20260812 07:07:37.964926 25219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:37.947346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54390 (local address 127.6.61.2:40553)
0812 07:07:37.947529 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:37.947534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:37.947616 (+ 82us) server_negotiation.cc:407] Connection header received
0812 07:07:37.947820 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:37.947825 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:37.947927 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:37.948025 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:37.949965 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:37.950854 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:37.952883 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:37.953342 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:37.953460 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:37.958673 (+ 5213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:37.958691 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:37.958710 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:37.958757 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:37.961929 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:37.962756 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:37.962766 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:37.962768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:37.962903 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:37.963354 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:37.963359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:37.963362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:37.963864 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:37.964123 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:37.964433 (+ 310us) server_negotiation.cc:299] Negotiation successful
0812 07:07:37.964622 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 07:07:37.965734 25037 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:07:38.065251 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 24895
I20260812 07:07:38.090474 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 25048
Aug 12 07:07:38 dist-test-slave-69rg krb5kdc[24621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.3@KRBTEST.COM
I20260812 07:07:38.107669 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:38.091423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37234 (local address 127.6.61.3:46821)
0812 07:07:38.091638 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:38.091645 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:38.091667 (+ 22us) server_negotiation.cc:407] Connection header received
0812 07:07:38.091897 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:38.091902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:38.091974 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:38.092126 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:38.093476 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:38.094323 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:38.095677 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:38.096081 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:38.096230 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:38.101161 (+ 4931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:38.101178 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:38.101196 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:38.101238 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:38.104241 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:38.105105 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:38.105114 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:38.105117 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:38.105307 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:38.105765 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:38.105770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:38.105772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:38.106252 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:38.106507 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:38.107073 (+ 566us) server_negotiation.cc:299] Negotiation successful
0812 07:07:38.107290 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 07:07:38.108422 25192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:07:38.198704 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 25048
I20260812 07:07:38.221938 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 24636
I20260812 07:07:38.223237 24708 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:07:38.339448 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 24636
I20260812 07:07:58.367079 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:39783
--webserver_interface=127.6.61.62
--webserver_port=33503
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:46191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:39783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:07:58.718070 25259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:07:58.718472 25259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:07:58.718650 25259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:07:58.725979 25259 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 07:07:58.726128 25259 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 07:07:58.726235 25259 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 07:07:58.726336 25259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:07:58.726419 25259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:07:58.726545 25259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:07:58.726635 25259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:07:58.726717 25259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:07:58.736608 25259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:46191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:39783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:39783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=33503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:07:58.737987 25259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:07:58.739380 25259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:07:58.764431 25267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:58.764462 25271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:58.764384 25268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:07:58.764910 25259 server_base.cc:1034] running on GCE node
I20260812 07:07:58.766235 25259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:07:58.767956 25259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:07:58.769196 25259 hybrid_clock.cc:648] HybridClock initialized: now 1786518478769120 us; error 46 us; skew 500 ppm
I20260812 07:07:58.769611 25259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:07:58 dist-test-slave-69rg krb5kdc[24621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518478, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:07:58.778563 25259 init.cc:377] Logged in from keytab as kudu/127.6.61.62@KRBTEST.COM (short username kudu)
I20260812 07:07:58.780463 25259 webserver.cc:460] Webserver started at http://127.6.61.62:33503/ using document root <none> and password file <none>
I20260812 07:07:58.781318 25259 fs_manager.cc:359] Metadata directory not provided
I20260812 07:07:58.781483 25259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:07:58.787804 25259 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 07:07:58.790853 25278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:58.792189 25259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 07:07:58.792384 25259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "cf960fc0f93f47f4bbb12e9580b94392"
format_stamp: "Formatted at 2026-08-12 07:07:35 on dist-test-slave-69rg"
server_key: "bce82cc2ad292df4a17751b72e88e4cf"
server_key_iv: "43b5eac11b3a46d174730a9400f0ad99"
server_key_version: "encryptionkey@0"
I20260812 07:07:58.793473 25259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:07:58.807749 25259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:07:58.812110 25259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:07:58.812422 25259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:07:58.833629 25259 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:39783
I20260812 07:07:58.833683 25338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:39783 every 8 connection(s)
I20260812 07:07:58.835438 25259 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:07:58.839579 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 25259
I20260812 07:07:58.840545 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.1:35167
--local_ip_for_outbound_sockets=127.6.61.1
--tserver_master_addrs=127.6.61.62:39783
--webserver_port=41937
--webserver_interface=127.6.61.1
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:46191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:07:58.843293 25339 sys_catalog.cc:263] Verifying existing consensus state
I20260812 07:07:58.846805 25339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392: Bootstrap starting.
I20260812 07:07:58.867271 25339 log.cc:824] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392: Log is configured to *not* fsync() on all Append() calls
I20260812 07:07:58.880079 25339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 07:07:58.880694 25339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392: Bootstrap complete.
I20260812 07:07:58.890305 25339 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf960fc0f93f47f4bbb12e9580b94392" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39783 } }
I20260812 07:07:58.891251 25339 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf960fc0f93f47f4bbb12e9580b94392, State: Initialized, Role: FOLLOWER
I20260812 07:07:58.891979 25339 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf960fc0f93f47f4bbb12e9580b94392" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39783 } }
I20260812 07:07:58.892268 25339 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:07:58.892419 25339 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:07:58.892608 25339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [term 1 FOLLOWER]: Advancing to term 2
I20260812 07:07:58.896771 25339 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf960fc0f93f47f4bbb12e9580b94392" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39783 } }
I20260812 07:07:58.897429 25339 leader_election.cc:302] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf960fc0f93f47f4bbb12e9580b94392; no voters:
I20260812 07:07:58.898238 25339 leader_election.cc:288] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 07:07:58.898372 25343 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 07:07:58.901345 25339 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:07:58.902156 25343 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [term 2 LEADER]: Becoming Leader. State: Replica: cf960fc0f93f47f4bbb12e9580b94392, State: Running, Role: LEADER
I20260812 07:07:58.902875 25343 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf960fc0f93f47f4bbb12e9580b94392" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39783 } }
I20260812 07:07:58.906399 25347 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf960fc0f93f47f4bbb12e9580b94392. Latest consensus state: current_term: 2 leader_uuid: "cf960fc0f93f47f4bbb12e9580b94392" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf960fc0f93f47f4bbb12e9580b94392" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39783 } } }
I20260812 07:07:58.906885 25347 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [sys.catalog]: This master's current role is: LEADER
I20260812 07:07:58.907554 25345 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cf960fc0f93f47f4bbb12e9580b94392" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf960fc0f93f47f4bbb12e9580b94392" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39783 } } }
I20260812 07:07:58.907946 25345 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [sys.catalog]: This master's current role is: LEADER
I20260812 07:07:58.914405 25352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:07:58.923364 25352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:07:58.924974 25352 catalog_manager.cc:1259] Loaded cluster ID: 4f0975c0b8584f9f8832dc83df8f4dd9
I20260812 07:07:58.925137 25352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:07:58.929447 25352 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:07:58.931183 25352 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392: Loaded TSK: 0
I20260812 07:07:58.981441 25364 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 07:07:58.983733 25352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392: Generated new TSK 1
I20260812 07:07:58.984293 25352 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 07:07:59.228894 25341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:07:59.229239 25341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:07:59.229413 25341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:07:59.237155 25341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:07:59.237326 25341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:07:59.237504 25341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.1
I20260812 07:07:59.247629 25341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:46191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.1:35167
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.61.1
--webserver_port=41937
--tserver_master_addrs=127.6.61.62:39783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.1
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:07:59.249004 25341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:07:59.250406 25341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:07:59.265056 25370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:59.265090 25371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:59.265516 25373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:07:59.266206 25341 server_base.cc:1034] running on GCE node
I20260812 07:07:59.266966 25341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:07:59.268152 25341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:07:59.269429 25341 hybrid_clock.cc:648] HybridClock initialized: now 1786518479269360 us; error 56 us; skew 500 ppm
I20260812 07:07:59.269806 25341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:07:59 dist-test-slave-69rg krb5kdc[24621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518479, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:07:59.274101 25341 init.cc:377] Logged in from keytab as kudu/127.6.61.1@KRBTEST.COM (short username kudu)
I20260812 07:07:59.275838 25341 webserver.cc:460] Webserver started at http://127.6.61.1:41937/ using document root <none> and password file <none>
I20260812 07:07:59.276557 25341 fs_manager.cc:359] Metadata directory not provided
I20260812 07:07:59.276691 25341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:07:59.282954 25341 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 07:07:59.286216 25380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:59.287672 25341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 07:07:59.287858 25341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/wal
uuid: "b3aad6d4634947aca8c854dad3fb9401"
format_stamp: "Formatted at 2026-08-12 07:07:36 on dist-test-slave-69rg"
server_key: "1726941850eaa84ad8046bd670747d3b"
server_key_iv: "c522ff3207f99a1056bcea945dc9bb0e"
server_key_version: "encryptionkey@0"
I20260812 07:07:59.288874 25341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:07:59.329676 25341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:07:59.334405 25341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:07:59.334712 25341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:07:59.335772 25341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:07:59.337152 25341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:07:59.337296 25341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:59.337426 25341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:07:59.337517 25341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:59.371639 25341 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.1:35167
I20260812 07:07:59.371737 25517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.1:35167 every 8 connection(s)
I20260812 07:07:59.373138 25341 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
I20260812 07:07:59.383453 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 25341
I20260812 07:07:59.384214 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.2:40553
--local_ip_for_outbound_sockets=127.6.61.2
--tserver_master_addrs=127.6.61.62:39783
--webserver_port=46073
--webserver_interface=127.6.61.2
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:46191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 07:07:59 dist-test-slave-69rg krb5kdc[24621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518479, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:07:59.397126 25521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:59.376301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.1:52037)
0812 07:07:59.377136 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:59.377180 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:59.378991 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:59.380381 (+ 1390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:59.380388 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:59.380904 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:07:59.381483 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:59.381501 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:59.383129 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:59.383137 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:59.384557 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:59.384571 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:59.385605 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:59.385612 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:59.385919 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:59.386615 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:07:59.386639 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:07:59.388822 (+ 2183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:07:59.392459 (+ 3637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:59.392469 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:59.392485 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:59.393024 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:59.393558 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:59.393563 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:59.393567 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:59.393761 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:59.394557 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:07:59.394564 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:07:59.394901 (+ 337us) client_negotiation.cc:769] Sending connection context
0812 07:07:59.395169 (+ 268us) client_negotiation.cc:240] Negotiation successful
0812 07:07:59.395749 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":371,"threads_started":1}
I20260812 07:07:59.397132 25520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:59.376195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:52037 (local address 127.6.61.62:39783)
0812 07:07:59.376872 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:59.376890 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:59.377328 (+ 438us) server_negotiation.cc:407] Connection header received
0812 07:07:59.379478 (+ 2150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:59.379490 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:59.379911 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:59.380213 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:59.381656 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:59.382953 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:59.384706 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:59.385360 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:59.385566 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:59.389102 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:59.389110 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:59.389125 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:59.389160 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:59.392311 (+ 3151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:59.393184 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:59.393191 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:59.393196 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:59.393442 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:59.393900 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:59.393906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:59.393910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:59.394406 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:59.394615 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:59.395397 (+ 782us) server_negotiation.cc:299] Negotiation successful
0812 07:07:59.395795 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":249,"threads_started":1}
I20260812 07:07:59.400360 25518 heartbeater.cc:347] Connected to a master server at 127.6.61.62:39783
I20260812 07:07:59.400722 25518 heartbeater.cc:464] Registering TS with master...
I20260812 07:07:59.401734 25518 heartbeater.cc:511] Master 127.6.61.62:39783 requested a full tablet report, sending...
I20260812 07:07:59.404268 25297 ts_manager.cc:194] Registered new tserver with Master: b3aad6d4634947aca8c854dad3fb9401 (127.6.61.1:35167)
I20260812 07:07:59.405651 25297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.1@KRBTEST.COM'} at 127.6.61.1:52037
W20260812 07:07:59.736145 25523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:07:59.736446 25523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:07:59.736605 25523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:07:59.743969 25523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:07:59.744076 25523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:07:59.744266 25523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.2
I20260812 07:07:59.754079 25523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:46191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.2:40553
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.61.2
--webserver_port=46073
--tserver_master_addrs=127.6.61.62:39783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.2
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:07:59.755429 25523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:07:59.756800 25523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:07:59.770061 25528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:59.770102 25529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:59.770654 25531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:07:59.771070 25523 server_base.cc:1034] running on GCE node
I20260812 07:07:59.771659 25523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:07:59.772594 25523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:07:59.773880 25523 hybrid_clock.cc:648] HybridClock initialized: now 1786518479773783 us; error 38 us; skew 500 ppm
I20260812 07:07:59.774214 25523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:07:59 dist-test-slave-69rg krb5kdc[24621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518479, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:07:59.777469 25523 init.cc:377] Logged in from keytab as kudu/127.6.61.2@KRBTEST.COM (short username kudu)
I20260812 07:07:59.779168 25523 webserver.cc:460] Webserver started at http://127.6.61.2:46073/ using document root <none> and password file <none>
I20260812 07:07:59.779871 25523 fs_manager.cc:359] Metadata directory not provided
I20260812 07:07:59.780042 25523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:07:59.785893 25523 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:07:59.788616 25539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:59.790062 25523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:07:59.790246 25523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/wal
uuid: "49e9a79960254210b48fe8468c593d01"
format_stamp: "Formatted at 2026-08-12 07:07:36 on dist-test-slave-69rg"
server_key: "e8ebd3b0fcc47f2792330901dea24982"
server_key_iv: "151e547d0e650c7cbc6361fb9a5ef693"
server_key_version: "encryptionkey@0"
I20260812 07:07:59.791266 25523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:07:59.834937 25523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:07:59.839588 25523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:07:59.839900 25523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:07:59.840951 25523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:07:59.842427 25523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:07:59.842555 25523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:59.842701 25523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:07:59.842794 25523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:59.879478 25523 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.2:40553
I20260812 07:07:59.879578 25673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.2:40553 every 8 connection(s)
I20260812 07:07:59.881017 25523 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
I20260812 07:07:59.889978 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 25523
I20260812 07:07:59.890865 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.3:46821
--local_ip_for_outbound_sockets=127.6.61.3
--tserver_master_addrs=127.6.61.62:39783
--webserver_port=36949
--webserver_interface=127.6.61.3
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:46191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 07:07:59 dist-test-slave-69rg krb5kdc[24621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518479, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:07:59.904788 25520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:59.883722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:46571 (local address 127.6.61.62:39783)
0812 07:07:59.883924 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:59.883928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:59.884987 (+ 1059us) server_negotiation.cc:407] Connection header received
0812 07:07:59.886906 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:59.886912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:59.887000 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:59.887094 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:59.888654 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:59.889413 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:59.891960 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:59.892496 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:59.892611 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:59.897357 (+ 4746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:59.897363 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:59.897366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:59.897428 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:59.901184 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:59.902172 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:59.902180 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:59.902186 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:59.902357 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:59.902883 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:59.902891 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:59.902897 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:59.903320 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:59.903506 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:59.904325 (+ 819us) server_negotiation.cc:299] Negotiation successful
0812 07:07:59.904533 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 07:07:59.906421 25676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:59.884079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.2:46571)
0812 07:07:59.884813 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:59.884851 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:59.886628 (+ 1777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:59.887391 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:59.887399 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:59.887911 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:07:59.888479 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:59.888496 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:59.889612 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:59.889617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:59.891776 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:59.891798 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:59.892761 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:59.892770 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:59.893151 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:59.893959 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:07:59.893994 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:07:59.897086 (+ 3092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:07:59.901411 (+ 4325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:59.901418 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:59.901432 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:59.902029 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:59.902516 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:59.902521 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:59.902523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:59.902775 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:59.903459 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:07:59.903466 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:07:59.903811 (+ 345us) client_negotiation.cc:769] Sending connection context
0812 07:07:59.904176 (+ 365us) client_negotiation.cc:240] Negotiation successful
0812 07:07:59.904815 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":297,"threads_started":1}
I20260812 07:07:59.908171 25674 heartbeater.cc:347] Connected to a master server at 127.6.61.62:39783
I20260812 07:07:59.908560 25674 heartbeater.cc:464] Registering TS with master...
I20260812 07:07:59.909343 25674 heartbeater.cc:511] Master 127.6.61.62:39783 requested a full tablet report, sending...
I20260812 07:07:59.910939 25297 ts_manager.cc:194] Registered new tserver with Master: 49e9a79960254210b48fe8468c593d01 (127.6.61.2:40553)
I20260812 07:07:59.911800 25297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.2@KRBTEST.COM'} at 127.6.61.2:46571
W20260812 07:08:00.248575 25677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:08:00.248868 25677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:08:00.249032 25677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:08:00.256388 25677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:08:00.256510 25677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:08:00.256697 25677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.3
I20260812 07:08:00.266752 25677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:46191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.3:46821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.61.3
--webserver_port=36949
--tserver_master_addrs=127.6.61.62:39783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.3
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:08:00.268116 25677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:08:00.269461 25677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:08:00.283131 25684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:00.283190 25685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:00.283215 25687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:08:00.284272 25677 server_base.cc:1034] running on GCE node
I20260812 07:08:00.284981 25677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:08:00.286034 25677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:08:00.287266 25677 hybrid_clock.cc:648] HybridClock initialized: now 1786518480287216 us; error 33 us; skew 500 ppm
I20260812 07:08:00.287568 25677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:08:00 dist-test-slave-69rg krb5kdc[24621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518480, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:08:00.291059 25677 init.cc:377] Logged in from keytab as kudu/127.6.61.3@KRBTEST.COM (short username kudu)
I20260812 07:08:00.292730 25677 webserver.cc:460] Webserver started at http://127.6.61.3:36949/ using document root <none> and password file <none>
I20260812 07:08:00.293546 25677 fs_manager.cc:359] Metadata directory not provided
I20260812 07:08:00.293717 25677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:08:00.299940 25677 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 07:08:00.303058 25695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:00.304418 25677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 07:08:00.304608 25677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/wal
uuid: "9639a9db44224890a8fcaaf1486080f5"
format_stamp: "Formatted at 2026-08-12 07:07:37 on dist-test-slave-69rg"
server_key: "bb48cd9d98f1e2087020ec6f5eac142c"
server_key_iv: "5653395901af7eb1d2e2234c552aed9e"
server_key_version: "encryptionkey@0"
I20260812 07:08:00.305704 25677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:08:00.330891 25677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:08:00.335075 25677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:08:00.335439 25677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:08:00.336539 25677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:08:00.337996 25677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:08:00.338145 25677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:00.338289 25677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:08:00.338404 25677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:00.373548 25677 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.3:46821
I20260812 07:08:00.373613 25829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.3:46821 every 8 connection(s)
I20260812 07:08:00.375074 25677 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
I20260812 07:08:00.376519 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 25677
Aug 12 07:08:00 dist-test-slave-69rg krb5kdc[24621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518480, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:08:00.396032 25520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:00.377738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38590 (local address 127.6.61.62:39783)
0812 07:08:00.378033 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:00.378043 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:00.378679 (+ 636us) server_negotiation.cc:407] Connection header received
0812 07:08:00.378905 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:00.378911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:00.379000 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:00.379123 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:00.382228 (+ 3105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:00.383411 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:00.385182 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:00.385756 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:00.385947 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:00.389451 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:00.389458 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:00.389463 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:08:00.389503 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:00.392698 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:00.393680 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:00.393686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:00.393690 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:00.393896 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:00.394310 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:00.394315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:00.394318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:00.394627 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:00.394935 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:00.395297 (+ 362us) server_negotiation.cc:299] Negotiation successful
0812 07:08:00.395556 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260812 07:08:00.399370 25834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:00.378687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:33505 (local address 127.6.61.62:39783)
0812 07:08:00.382308 (+ 3621us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:00.382313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:00.382333 (+ 20us) server_negotiation.cc:407] Connection header received
0812 07:08:00.382514 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:00.382518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:00.382586 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:00.382730 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:00.384431 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:00.385419 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:00.387247 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:00.388015 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:00.388167 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:00.392232 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:00.392243 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:00.392247 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:08:00.392312 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:00.395599 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:00.396592 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:00.396596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:00.396598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:00.396726 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:00.397218 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:00.397223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:00.397228 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:00.397571 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:00.397712 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:00.398707 (+ 995us) server_negotiation.cc:299] Negotiation successful
0812 07:08:00.398942 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3525,"thread_start_us":3351,"threads_started":1}
I20260812 07:08:00.400338 25835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:00.379347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.3:33505)
0812 07:08:00.380222 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:00.380269 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:00.382231 (+ 1962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:00.382981 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:00.382992 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:00.383700 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:08:00.384242 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:00.384259 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:00.385600 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:00.385606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:00.387089 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:00.387103 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:00.388227 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:00.388232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:00.388423 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:00.389026 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:08:00.389048 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:08:00.391939 (+ 2891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:08:00.395759 (+ 3820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:08:00.395768 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:08:00.395785 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:08:00.396414 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:08:00.396846 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:08:00.396851 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:08:00.396855 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:08:00.397054 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:08:00.397753 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:08:00.397762 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:08:00.398244 (+ 482us) client_negotiation.cc:769] Sending connection context
0812 07:08:00.398588 (+ 344us) client_negotiation.cc:240] Negotiation successful
0812 07:08:00.399116 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":359,"threads_started":1}
I20260812 07:08:00.401691 25830 heartbeater.cc:347] Connected to a master server at 127.6.61.62:39783
I20260812 07:08:00.402081 25830 heartbeater.cc:464] Registering TS with master...
I20260812 07:08:00.402830 25830 heartbeater.cc:511] Master 127.6.61.62:39783 requested a full tablet report, sending...
I20260812 07:08:00.404163 25297 ts_manager.cc:194] Registered new tserver with Master: 9639a9db44224890a8fcaaf1486080f5 (127.6.61.3:46821)
I20260812 07:08:00.404996 25297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.3@KRBTEST.COM'} at 127.6.61.3:33505
I20260812 07:08:00.409657 25518 heartbeater.cc:503] Master 127.6.61.62:39783 was elected leader, sending a full tablet report...
I20260812 07:08:00.411774 6388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 07:08:00.418793 25834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:00.412786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38606 (local address 127.6.61.62:39783)
0812 07:08:00.412979 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:00.412987 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:00.413602 (+ 615us) server_negotiation.cc:407] Connection header received
0812 07:08:00.413808 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:00.413862 (+ 54us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:00.414007 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:00.414125 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:08:00.414564 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:00.415328 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:00.416975 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:00.417410 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:00.417509 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:00.418003 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:08:00.418230 (+ 227us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0812 07:08:00.418525 (+ 295us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38606: token expired
Metrics: {"server-negotiator.queue_time_us":61}
W20260812 07:08:00.418937 25834 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38606: token expired
W20260812 07:08:00.419530 25205 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.6.61.62:39783)
I20260812 07:08:00.432675 25834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:00.420091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38622 (local address 127.6.61.62:39783)
0812 07:08:00.420355 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:00.420360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:00.420376 (+ 16us) server_negotiation.cc:407] Connection header received
0812 07:08:00.420418 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:00.420423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:00.420481 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:00.420568 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:00.421957 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:00.422750 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:00.424086 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:00.424545 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:00.424648 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:00.427459 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:00.427465 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:00.427468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:08:00.427502 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:00.430323 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:00.431011 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:00.431015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:00.431017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:00.431135 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:00.431481 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:00.431484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:00.431486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:00.431798 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:00.432007 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:00.432260 (+ 253us) server_negotiation.cc:299] Negotiation successful
0812 07:08:00.432424 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260812 07:08:00.436388 25296 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:08:00.443867 25296 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:08:00.476871 25846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:00.456174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:35167 (local address 127.0.0.1:37916)
0812 07:08:00.465293 (+ 9119us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:00.465364 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:00.465602 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:00.467337 (+ 1735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:00.467349 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:00.467398 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:08:00.467979 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:00.467996 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:00.471681 (+ 3685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:00.471690 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:00.474387 (+ 2697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:00.474404 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:00.475770 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:00.475776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:00.475995 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:00.476061 (+ 66us) client_negotiation.cc:769] Sending connection context
0812 07:08:00.476211 (+ 150us) client_negotiation.cc:240] Negotiation successful
0812 07:08:00.476448 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8740,"thread_start_us":13676,"threads_started":1}
I20260812 07:08:00.477002 25845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:00.456506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37916 (local address 127.6.61.1:35167)
0812 07:08:00.466561 (+ 10055us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:00.466573 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:00.466613 (+ 40us) server_negotiation.cc:407] Connection header received
0812 07:08:00.466766 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:00.466779 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:00.467087 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:00.467340 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:08:00.469269 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:00.471447 (+ 2178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:00.474576 (+ 3129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:00.475541 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:00.475711 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:00.475973 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:00.476455 (+ 482us) server_negotiation.cc:299] Negotiation successful
0812 07:08:00.476649 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9894,"thread_start_us":9767,"threads_started":1}
I20260812 07:08:00.481257 25843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:00.456205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:40553 (local address 127.0.0.1:42498)
0812 07:08:00.469591 (+ 13386us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:00.469621 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:00.469774 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:00.473422 (+ 3648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:00.473428 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:00.473457 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:08:00.473975 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:00.473992 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:00.476262 (+ 2270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:00.476267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:00.478867 (+ 2600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:00.478880 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:00.480426 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:00.480432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:00.480610 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:00.480643 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 07:08:00.480735 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 07:08:00.480863 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13224,"mutex_wait_us":547,"thread_start_us":15527,"threads_started":1}
I20260812 07:08:00.481369 25842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:00.456224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42498 (local address 127.6.61.2:40553)
0812 07:08:00.472865 (+ 16641us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:00.472873 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:00.472898 (+ 25us) server_negotiation.cc:407] Connection header received
0812 07:08:00.472983 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:00.472991 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:00.473214 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:00.474167 (+ 953us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:08:00.474433 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:00.476060 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:00.479027 (+ 2967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:00.480172 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:00.480333 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:00.480525 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:00.480824 (+ 299us) server_negotiation.cc:299] Negotiation successful
0812 07:08:00.480999 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16487,"thread_start_us":16371,"threads_started":1}
I20260812 07:08:00.482282 25844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:00.459110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:46821 (local address 127.0.0.1:45310)
0812 07:08:00.471766 (+ 12656us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:00.471790 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:00.471935 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:00.473280 (+ 1345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:00.473285 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:00.473315 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:08:00.473898 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:00.473914 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:00.476262 (+ 2348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:00.476267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:00.479586 (+ 3319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:00.479602 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:00.481426 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:00.481436 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:00.481625 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:00.481666 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 07:08:00.481751 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 07:08:00.481911 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12456,"mutex_wait_us":70,"thread_start_us":5576,"threads_started":1}
I20260812 07:08:00.481639 25434 tablet_service.cc:1467] Processing CreateTablet for tablet 69062ea8e2714cf8b639da06c2edc1c8 (DEFAULT_TABLE table=test-table [id=6eaece54982e48668d98cd738619a0c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:08:00.483237 25847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:00.459899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45310 (local address 127.6.61.3:46821)
0812 07:08:00.472615 (+ 12716us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:00.472627 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:00.472665 (+ 38us) server_negotiation.cc:407] Connection header received
0812 07:08:00.472780 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:00.472788 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:00.473019 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:00.474131 (+ 1112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:08:00.474441 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:00.476099 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:00.479778 (+ 3679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:00.481179 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:00.482322 (+ 1143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:00.482570 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:00.482663 (+ 93us) server_negotiation.cc:299] Negotiation successful
0812 07:08:00.482830 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12522,"thread_start_us":12339,"threads_started":1}
I20260812 07:08:00.484046 25434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69062ea8e2714cf8b639da06c2edc1c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:08:00.485361 25598 tablet_service.cc:1467] Processing CreateTablet for tablet 69062ea8e2714cf8b639da06c2edc1c8 (DEFAULT_TABLE table=test-table [id=6eaece54982e48668d98cd738619a0c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:08:00.485996 25752 tablet_service.cc:1467] Processing CreateTablet for tablet 69062ea8e2714cf8b639da06c2edc1c8 (DEFAULT_TABLE table=test-table [id=6eaece54982e48668d98cd738619a0c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:08:00.486747 25598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69062ea8e2714cf8b639da06c2edc1c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:08:00.487226 25752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69062ea8e2714cf8b639da06c2edc1c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:08:00.499768 25848 tablet_bootstrap.cc:492] T 69062ea8e2714cf8b639da06c2edc1c8 P 9639a9db44224890a8fcaaf1486080f5: Bootstrap starting.
I20260812 07:08:00.504171 25848 tablet_bootstrap.cc:654] T 69062ea8e2714cf8b639da06c2edc1c8 P 9639a9db44224890a8fcaaf1486080f5: Neither blocks nor log segments found. Creating new log.
I20260812 07:08:00.505268 25848 log.cc:824] T 69062ea8e2714cf8b639da06c2edc1c8 P 9639a9db44224890a8fcaaf1486080f5: Log is configured to *not* fsync() on all Append() calls
I20260812 07:08:00.506332 25849 tablet_bootstrap.cc:492] T 69062ea8e2714cf8b639da06c2edc1c8 P b3aad6d4634947aca8c854dad3fb9401: Bootstrap starting.
I20260812 07:08:00.509230 25850 tablet_bootstrap.cc:492] T 69062ea8e2714cf8b639da06c2edc1c8 P 49e9a79960254210b48fe8468c593d01: Bootstrap starting.
I20260812 07:08:00.510558 25848 tablet_bootstrap.cc:492] T 69062ea8e2714cf8b639da06c2edc1c8 P 9639a9db44224890a8fcaaf1486080f5: No bootstrap required, opened a new log
I20260812 07:08:00.510891 25849 tablet_bootstrap.cc:654] T 69062ea8e2714cf8b639da06c2edc1c8 P b3aad6d4634947aca8c854dad3fb9401: Neither blocks nor log segments found. Creating new log.
I20260812 07:08:00.510921 25848 ts_tablet_manager.cc:1397] T 69062ea8e2714cf8b639da06c2edc1c8 P 9639a9db44224890a8fcaaf1486080f5: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260812 07:08:00.512238 25849 log.cc:824] T 69062ea8e2714cf8b639da06c2edc1c8 P b3aad6d4634947aca8c854dad3fb9401: Log is configured to *not* fsync() on all Append() calls
I20260812 07:08:00.513069 25850 tablet_bootstrap.cc:654] T 69062ea8e2714cf8b639da06c2edc1c8 P 49e9a79960254210b48fe8468c593d01: Neither blocks nor log segments found. Creating new log.
I20260812 07:08:00.514281 25850 log.cc:824] T 69062ea8e2714cf8b639da06c2edc1c8 P 49e9a79960254210b48fe8468c593d01: Log is configured to *not* fsync() on all Append() calls
I20260812 07:08:00.516750 25849 tablet_bootstrap.cc:492] T 69062ea8e2714cf8b639da06c2edc1c8 P b3aad6d4634947aca8c854dad3fb9401: No bootstrap required, opened a new log
I20260812 07:08:00.517027 25849 ts_tablet_manager.cc:1397] T 69062ea8e2714cf8b639da06c2edc1c8 P b3aad6d4634947aca8c854dad3fb9401: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260812 07:08:00.518648 25850 tablet_bootstrap.cc:492] T 69062ea8e2714cf8b639da06c2edc1c8 P 49e9a79960254210b48fe8468c593d01: No bootstrap required, opened a new log
I20260812 07:08:00.519011 25850 ts_tablet_manager.cc:1397] T 69062ea8e2714cf8b639da06c2edc1c8 P 49e9a79960254210b48fe8468c593d01: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20260812 07:08:00.522022 25848 raft_consensus.cc:348] T 69062ea8e2714cf8b639da06c2edc1c8 P 9639a9db44224890a8fcaaf1486080f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e9a79960254210b48fe8468c593d01" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 40553 } } peers { permanent_uuid: "b3aad6d4634947aca8c854dad3fb9401" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 35167 } } peers { permanent_uuid: "9639a9db44224890a8fcaaf1486080f5" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 46821 } }
I20260812 07:08:00.522481 25848 raft_consensus.cc:374] T 69062ea8e2714cf8b639da06c2edc1c8 P 9639a9db44224890a8fcaaf1486080f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:08:00.522626 25848 raft_consensus.cc:729] T 69062ea8e2714cf8b639da06c2edc1c8 P 9639a9db44224890a8fcaaf1486080f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9639a9db44224890a8fcaaf1486080f5, State: Initialized, Role: FOLLOWER
I20260812 07:08:00.523337 25848 consensus_queue.cc:260] T 69062ea8e2714cf8b639da06c2edc1c8 P 9639a9db44224890a8fcaaf1486080f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e9a79960254210b48fe8468c593d01" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 40553 } } peers { permanent_uuid: "b3aad6d4634947aca8c854dad3fb9401" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 35167 } } peers { permanent_uuid: "9639a9db44224890a8fcaaf1486080f5" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 46821 } }
I20260812 07:08:00.524487 25830 heartbeater.cc:503] Master 127.6.61.62:39783 was elected leader, sending a full tablet report...
I20260812 07:08:00.525158 25848 ts_tablet_manager.cc:1428] T 69062ea8e2714cf8b639da06c2edc1c8 P 9639a9db44224890a8fcaaf1486080f5: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260812 07:08:00.527841 25850 raft_consensus.cc:348] T 69062ea8e2714cf8b639da06c2edc1c8 P 49e9a79960254210b48fe8468c593d01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e9a79960254210b48fe8468c593d01" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 40553 } } peers { permanent_uuid: "b3aad6d4634947aca8c854dad3fb9401" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 35167 } } peers { permanent_uuid: "9639a9db44224890a8fcaaf1486080f5" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 46821 } }
I20260812 07:08:00.528304 25850 raft_consensus.cc:374] T 69062ea8e2714cf8b639da06c2edc1c8 P 49e9a79960254210b48fe8468c593d01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:08:00.528064 25849 raft_consensus.cc:348] T 69062ea8e2714cf8b639da06c2edc1c8 P b3aad6d4634947aca8c854dad3fb9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e9a79960254210b48fe8468c593d01" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 40553 } } peers { permanent_uuid: "b3aad6d4634947aca8c854dad3fb9401" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 35167 } } peers { permanent_uuid: "9639a9db44224890a8fcaaf1486080f5" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 46821 } }
I20260812 07:08:00.528441 25850 raft_consensus.cc:729] T 69062ea8e2714cf8b639da06c2edc1c8 P 49e9a79960254210b48fe8468c593d01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e9a79960254210b48fe8468c593d01, State: Initialized, Role: FOLLOWER
I20260812 07:08:00.528579 25849 raft_consensus.cc:374] T 69062ea8e2714cf8b639da06c2edc1c8 P b3aad6d4634947aca8c854dad3fb9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:08:00.528716 25849 raft_consensus.cc:729] T 69062ea8e2714cf8b639da06c2edc1c8 P b3aad6d4634947aca8c854dad3fb9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3aad6d4634947aca8c854dad3fb9401, State: Initialized, Role: FOLLOWER
I20260812 07:08:00.529062 25850 consensus_queue.cc:260] T 69062ea8e2714cf8b639da06c2edc1c8 P 49e9a79960254210b48fe8468c593d01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e9a79960254210b48fe8468c593d01" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 40553 } } peers { permanent_uuid: "b3aad6d4634947aca8c854dad3fb9401" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 35167 } } peers { permanent_uuid: "9639a9db44224890a8fcaaf1486080f5" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 46821 } }
I20260812 07:08:00.529311 25849 consensus_queue.cc:260] T 69062ea8e2714cf8b639da06c2edc1c8 P b3aad6d4634947aca8c854dad3fb9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e9a79960254210b48fe8468c593d01" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 40553 } } peers { permanent_uuid: "b3aad6d4634947aca8c854dad3fb9401" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 35167 } } peers { permanent_uuid: "9639a9db44224890a8fcaaf1486080f5" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 46821 } }
I20260812 07:08:00.531838 25674 heartbeater.cc:503] Master 127.6.61.62:39783 was elected leader, sending a full tablet report...
I20260812 07:08:00.532503 25850 ts_tablet_manager.cc:1428] T 69062ea8e2714cf8b639da06c2edc1c8 P 49e9a79960254210b48fe8468c593d01: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260812 07:08:00.533156 25849 ts_tablet_manager.cc:1428] T 69062ea8e2714cf8b639da06c2edc1c8 P b3aad6d4634947aca8c854dad3fb9401: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
W20260812 07:08:00.625782 25519 tablet.cc:2369] T 69062ea8e2714cf8b639da06c2edc1c8 P b3aad6d4634947aca8c854dad3fb9401: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 07:08:00.627672 25831 tablet.cc:2369] T 69062ea8e2714cf8b639da06c2edc1c8 P 9639a9db44224890a8fcaaf1486080f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 07:08:00.633118 25675 tablet.cc:2369] T 69062ea8e2714cf8b639da06c2edc1c8 P 49e9a79960254210b48fe8468c593d01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:08:00.743619 25855 raft_consensus.cc:482] T 69062ea8e2714cf8b639da06c2edc1c8 P 49e9a79960254210b48fe8468c593d01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:08:00.743963 25855 raft_consensus.cc:504] T 69062ea8e2714cf8b639da06c2edc1c8 P 49e9a79960254210b48fe8468c593d01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e9a79960254210b48fe8468c593d01" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 40553 } } peers { permanent_uuid: "b3aad6d4634947aca8c854dad3fb9401" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 35167 } } peers { permanent_uuid: "9639a9db44224890a8fcaaf1486080f5" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 46821 } }
I20260812 07:08:00.745795 25855 leader_election.cc:288] T 69062ea8e2714cf8b639da06c2edc1c8 P 49e9a79960254210b48fe8468c593d01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3aad6d4634947aca8c854dad3fb9401 (127.6.61.1:35167), 9639a9db44224890a8fcaaf1486080f5 (127.6.61.3:46821)
I20260812 07:08:00.753532 25847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:00.746332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:44603 (local address 127.6.61.3:46821)
0812 07:08:00.746582 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:00.746590 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:00.747715 (+ 1125us) server_negotiation.cc:407] Connection header received
0812 07:08:00.747924 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:00.747929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:00.748005 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:00.748109 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:08:00.748804 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:00.749527 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:00.751671 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:00.752494 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:00.752662 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:00.752795 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:00.753107 (+ 312us) server_negotiation.cc:299] Negotiation successful
0812 07:08:00.753248 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 07:08:00.753629 25845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:00.746247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:56517 (local address 127.6.61.1:35167)
0812 07:08:00.746496 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:00.746506 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:00.747622 (+ 1116us) server_negotiation.cc:407] Connection header received
0812 07:08:00.747850 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:00.747856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:00.747927 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:00.748028 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:08:00.748617 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:00.749368 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:00.751689 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:00.752614 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:00.752786 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:00.752903 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:00.753221 (+ 318us) server_negotiation.cc:299] Negotiation successful
0812 07:08:00.753355 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 07:08:00.753736 25857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:00.746251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:46821 (local address 127.6.61.2:44603)
0812 07:08:00.747603 (+ 1352us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:00.747630 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:00.747768 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:00.748134 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:00.748138 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:00.748180 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:08:00.748637 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:00.748646 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:00.749679 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:00.749685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:00.751478 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:00.751491 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:00.752733 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:00.752739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:00.752934 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:00.752959 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 07:08:00.753074 (+ 115us) client_negotiation.cc:240] Negotiation successful
0812 07:08:00.753219 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1217,"mutex_wait_us":2,"thread_start_us":502,"threads_started":1}
I20260812 07:08:00.753793 25858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:00.746172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:35167 (local address 127.6.61.2:56517)
0812 07:08:00.747501 (+ 1329us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:00.747517 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:00.747668 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:00.748044 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:00.748048 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:00.748071 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:08:00.748457 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:00.748467 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:00.749513 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:00.749519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:00.751487 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:00.751502 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:00.752868 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:00.752874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:00.753044 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:00.753068 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 07:08:00.753188 (+ 120us) client_negotiation.cc:240] Negotiation successful
0812 07:08:00.753328 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1174,"mutex_wait_us":47,"thread_start_us":796,"threads_started":1}
I20260812 07:08:00.754987 25773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69062ea8e2714cf8b639da06c2edc1c8" candidate_uuid: "49e9a79960254210b48fe8468c593d01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9639a9db44224890a8fcaaf1486080f5" is_pre_election: true
I20260812 07:08:00.755023 25458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69062ea8e2714cf8b639da06c2edc1c8" candidate_uuid: "49e9a79960254210b48fe8468c593d01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3aad6d4634947aca8c854dad3fb9401" is_pre_election: true
I20260812 07:08:00.755549 25773 raft_consensus.cc:2454] T 69062ea8e2714cf8b639da06c2edc1c8 P 9639a9db44224890a8fcaaf1486080f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e9a79960254210b48fe8468c593d01 in term 0.
I20260812 07:08:00.755549 25458 raft_consensus.cc:2454] T 69062ea8e2714cf8b639da06c2edc1c8 P b3aad6d4634947aca8c854dad3fb9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e9a79960254210b48fe8468c593d01 in term 0.
I20260812 07:08:00.756290 25542 leader_election.cc:302] T 69062ea8e2714cf8b639da06c2edc1c8 P 49e9a79960254210b48fe8468c593d01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e9a79960254210b48fe8468c593d01, 9639a9db44224890a8fcaaf1486080f5; no voters:
I20260812 07:08:00.756742 25855 raft_consensus.cc:2781] T 69062ea8e2714cf8b639da06c2edc1c8 P 49e9a79960254210b48fe8468c593d01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:08:00.756922 25855 raft_consensus.cc:482] T 69062ea8e2714cf8b639da06c2edc1c8 P 49e9a79960254210b48fe8468c593d01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:08:00.757045 25855 raft_consensus.cc:3037] T 69062ea8e2714cf8b639da06c2edc1c8 P 49e9a79960254210b48fe8468c593d01 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:08:00.759377 25855 raft_consensus.cc:504] T 69062ea8e2714cf8b639da06c2edc1c8 P 49e9a79960254210b48fe8468c593d01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e9a79960254210b48fe8468c593d01" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 40553 } } peers { permanent_uuid: "b3aad6d4634947aca8c854dad3fb9401" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 35167 } } peers { permanent_uuid: "9639a9db44224890a8fcaaf1486080f5" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 46821 } }
I20260812 07:08:00.760020 25855 leader_election.cc:288] T 69062ea8e2714cf8b639da06c2edc1c8 P 49e9a79960254210b48fe8468c593d01 [CANDIDATE]: Term 1 election: Requested vote from peers b3aad6d4634947aca8c854dad3fb9401 (127.6.61.1:35167), 9639a9db44224890a8fcaaf1486080f5 (127.6.61.3:46821)
I20260812 07:08:00.760533 25458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69062ea8e2714cf8b639da06c2edc1c8" candidate_uuid: "49e9a79960254210b48fe8468c593d01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3aad6d4634947aca8c854dad3fb9401"
I20260812 07:08:00.760542 25773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69062ea8e2714cf8b639da06c2edc1c8" candidate_uuid: "49e9a79960254210b48fe8468c593d01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9639a9db44224890a8fcaaf1486080f5"
I20260812 07:08:00.760742 25773 raft_consensus.cc:3037] T 69062ea8e2714cf8b639da06c2edc1c8 P 9639a9db44224890a8fcaaf1486080f5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:08:00.760761 25458 raft_consensus.cc:3037] T 69062ea8e2714cf8b639da06c2edc1c8 P b3aad6d4634947aca8c854dad3fb9401 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:08:00.763204 25458 raft_consensus.cc:2454] T 69062ea8e2714cf8b639da06c2edc1c8 P b3aad6d4634947aca8c854dad3fb9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e9a79960254210b48fe8468c593d01 in term 1.
I20260812 07:08:00.763216 25773 raft_consensus.cc:2454] T 69062ea8e2714cf8b639da06c2edc1c8 P 9639a9db44224890a8fcaaf1486080f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e9a79960254210b48fe8468c593d01 in term 1.
I20260812 07:08:00.763691 25540 leader_election.cc:302] T 69062ea8e2714cf8b639da06c2edc1c8 P 49e9a79960254210b48fe8468c593d01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e9a79960254210b48fe8468c593d01, b3aad6d4634947aca8c854dad3fb9401; no voters:
I20260812 07:08:00.764077 25855 raft_consensus.cc:2781] T 69062ea8e2714cf8b639da06c2edc1c8 P 49e9a79960254210b48fe8468c593d01 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:08:00.764684 25855 raft_consensus.cc:686] T 69062ea8e2714cf8b639da06c2edc1c8 P 49e9a79960254210b48fe8468c593d01 [term 1 LEADER]: Becoming Leader. State: Replica: 49e9a79960254210b48fe8468c593d01, State: Running, Role: LEADER
I20260812 07:08:00.765192 25855 consensus_queue.cc:237] T 69062ea8e2714cf8b639da06c2edc1c8 P 49e9a79960254210b48fe8468c593d01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e9a79960254210b48fe8468c593d01" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 40553 } } peers { permanent_uuid: "b3aad6d4634947aca8c854dad3fb9401" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 35167 } } peers { permanent_uuid: "9639a9db44224890a8fcaaf1486080f5" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 46821 } }
I20260812 07:08:00.770335 25297 catalog_manager.cc:5495] T 69062ea8e2714cf8b639da06c2edc1c8 P 49e9a79960254210b48fe8468c593d01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49e9a79960254210b48fe8468c593d01 (127.6.61.2). New cstate: current_term: 1 leader_uuid: "49e9a79960254210b48fe8468c593d01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e9a79960254210b48fe8468c593d01" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 40553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3aad6d4634947aca8c854dad3fb9401" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 35167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9639a9db44224890a8fcaaf1486080f5" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 46821 } health_report { overall_health: UNKNOWN } } }
I20260812 07:08:00.851447 25842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:00.844438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42502 (local address 127.6.61.2:40553)
0812 07:08:00.844635 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:00.844641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:00.844905 (+ 264us) server_negotiation.cc:407] Connection header received
0812 07:08:00.845129 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:00.845133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:00.845218 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:00.845356 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:08:00.845791 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:00.846615 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:00.848739 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:00.849186 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:00.849288 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:00.850222 (+ 934us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:08:00.850760 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:08:00.850877 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:00.851079 (+ 202us) server_negotiation.cc:299] Negotiation successful
0812 07:08:00.851214 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 07:08:00.860212 25773 raft_consensus.cc:1261] T 69062ea8e2714cf8b639da06c2edc1c8 P 9639a9db44224890a8fcaaf1486080f5 [term 1 FOLLOWER]: Refusing update from remote peer 49e9a79960254210b48fe8468c593d01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:08:00.860261 25458 raft_consensus.cc:1261] T 69062ea8e2714cf8b639da06c2edc1c8 P b3aad6d4634947aca8c854dad3fb9401 [term 1 FOLLOWER]: Refusing update from remote peer 49e9a79960254210b48fe8468c593d01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:08:00.861310 25859 consensus_queue.cc:1035] T 69062ea8e2714cf8b639da06c2edc1c8 P 49e9a79960254210b48fe8468c593d01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3aad6d4634947aca8c854dad3fb9401" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 35167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:08:00.861696 25855 consensus_queue.cc:1035] T 69062ea8e2714cf8b639da06c2edc1c8 P 49e9a79960254210b48fe8468c593d01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9639a9db44224890a8fcaaf1486080f5" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:08:00.871029 25863 mvcc.cc:205] Tried to move back new op lower bound from 7317579697592496128 to 7317579697218850816. Current Snapshot: MvccSnapshot[applied={T|T < 7317579697592496128}]
I20260812 07:08:00.873313 25865 mvcc.cc:205] Tried to move back new op lower bound from 7317579697592496128 to 7317579697218850816. Current Snapshot: MvccSnapshot[applied={T|T < 7317579697592496128}]
I20260812 07:08:00.895730 25297 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:08:00.896099 25297 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:08:00.900481 25297 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392: Sending DeleteTablet for 3 replicas of tablet 69062ea8e2714cf8b639da06c2edc1c8
I20260812 07:08:00.901621 25598 tablet_service.cc:1514] Processing DeleteTablet for tablet 69062ea8e2714cf8b639da06c2edc1c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:08:00 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:42498
I20260812 07:08:00.902089 25434 tablet_service.cc:1514] Processing DeleteTablet for tablet 69062ea8e2714cf8b639da06c2edc1c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:08:00 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:37916
I20260812 07:08:00.902284 25752 tablet_service.cc:1514] Processing DeleteTablet for tablet 69062ea8e2714cf8b639da06c2edc1c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:08:00 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:45310
I20260812 07:08:00.903637 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 25341
I20260812 07:08:00.912938 25876 tablet_replica.cc:330] T 69062ea8e2714cf8b639da06c2edc1c8 P 49e9a79960254210b48fe8468c593d01: stopping tablet replica
I20260812 07:08:00.915870 25877 tablet_replica.cc:330] T 69062ea8e2714cf8b639da06c2edc1c8 P b3aad6d4634947aca8c854dad3fb9401: stopping tablet replica
I20260812 07:08:00.916304 25877 raft_consensus.cc:2229] T 69062ea8e2714cf8b639da06c2edc1c8 P b3aad6d4634947aca8c854dad3fb9401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:08:00.916859 25877 raft_consensus.cc:2258] T 69062ea8e2714cf8b639da06c2edc1c8 P b3aad6d4634947aca8c854dad3fb9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:08:00.917110 25876 raft_consensus.cc:2229] T 69062ea8e2714cf8b639da06c2edc1c8 P 49e9a79960254210b48fe8468c593d01 [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:08:00.918256 25876 raft_consensus.cc:2258] T 69062ea8e2714cf8b639da06c2edc1c8 P 49e9a79960254210b48fe8468c593d01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:08:00.919798 25878 tablet_replica.cc:330] T 69062ea8e2714cf8b639da06c2edc1c8 P 9639a9db44224890a8fcaaf1486080f5: stopping tablet replica
I20260812 07:08:00.920238 25878 raft_consensus.cc:2229] T 69062ea8e2714cf8b639da06c2edc1c8 P 9639a9db44224890a8fcaaf1486080f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:08:00.920583 25876 ts_tablet_manager.cc:1905] T 69062ea8e2714cf8b639da06c2edc1c8 P 49e9a79960254210b48fe8468c593d01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:08:00.922806 25877 ts_tablet_manager.cc:1905] T 69062ea8e2714cf8b639da06c2edc1c8 P b3aad6d4634947aca8c854dad3fb9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:08:00.923161 25878 raft_consensus.cc:2258] T 69062ea8e2714cf8b639da06c2edc1c8 P 9639a9db44224890a8fcaaf1486080f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:08:00.928661 25876 ts_tablet_manager.cc:1918] T 69062ea8e2714cf8b639da06c2edc1c8 P 49e9a79960254210b48fe8468c593d01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:08:00.928923 25876 log.cc:1196] T 69062ea8e2714cf8b639da06c2edc1c8 P 49e9a79960254210b48fe8468c593d01: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/wal/wals/69062ea8e2714cf8b639da06c2edc1c8
I20260812 07:08:00.929597 25876 ts_tablet_manager.cc:1939] T 69062ea8e2714cf8b639da06c2edc1c8 P 49e9a79960254210b48fe8468c593d01: Deleting consensus metadata
I20260812 07:08:00.931035 25877 ts_tablet_manager.cc:1918] T 69062ea8e2714cf8b639da06c2edc1c8 P b3aad6d4634947aca8c854dad3fb9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:08:00.931674 25280 catalog_manager.cc:4841] TS 49e9a79960254210b48fe8468c593d01 (127.6.61.2:40553): tablet 69062ea8e2714cf8b639da06c2edc1c8 (table test-table [id=6eaece54982e48668d98cd738619a0c8]) successfully deleted
I20260812 07:08:00.932835 25878 ts_tablet_manager.cc:1905] T 69062ea8e2714cf8b639da06c2edc1c8 P 9639a9db44224890a8fcaaf1486080f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:08:00.938122 25878 ts_tablet_manager.cc:1918] T 69062ea8e2714cf8b639da06c2edc1c8 P 9639a9db44224890a8fcaaf1486080f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:08:00.933682 25845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:00.904686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37922 (local address 127.6.61.1:35167)
0812 07:08:00.904930 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:00.904940 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:00.912206 (+ 7266us) server_negotiation.cc:407] Connection header received
0812 07:08:00.913991 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:00.914002 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:00.914158 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:00.914258 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:00.916998 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:00.918222 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:00.920410 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:00.921148 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:00.921286 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:00.925249 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:00.925266 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:00.925286 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:08:00.925335 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:00.929703 (+ 4368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:00.930887 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:00.930897 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:00.930902 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:00.931085 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:00.931652 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:00.931657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:00.931660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:00.932142 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:00.932425 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:00.933054 (+ 629us) server_negotiation.cc:299] Negotiation successful
0812 07:08:00.933292 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 07:08:00.938305 25878 log.cc:1196] T 69062ea8e2714cf8b639da06c2edc1c8 P 9639a9db44224890a8fcaaf1486080f5: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/wal/wals/69062ea8e2714cf8b639da06c2edc1c8
I20260812 07:08:00.938874 25878 ts_tablet_manager.cc:1939] T 69062ea8e2714cf8b639da06c2edc1c8 P 9639a9db44224890a8fcaaf1486080f5: Deleting consensus metadata
I20260812 07:08:00.939000 25877 log.cc:1196] T 69062ea8e2714cf8b639da06c2edc1c8 P b3aad6d4634947aca8c854dad3fb9401: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/wal/wals/69062ea8e2714cf8b639da06c2edc1c8
I20260812 07:08:00.939153 25513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:08:00.941088 25281 catalog_manager.cc:4841] TS 9639a9db44224890a8fcaaf1486080f5 (127.6.61.3:46821): tablet 69062ea8e2714cf8b639da06c2edc1c8 (table test-table [id=6eaece54982e48668d98cd738619a0c8]) successfully deleted
I20260812 07:08:01.031010 25877 ts_tablet_manager.cc:1939] T 69062ea8e2714cf8b639da06c2edc1c8 P b3aad6d4634947aca8c854dad3fb9401: Deleting consensus metadata
I20260812 07:08:01.031842 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 25341
I20260812 07:08:01.057049 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 25523
W20260812 07:08:01.057072 25279 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.61.1:35167 (error 108)
I20260812 07:08:01.059235 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:01.058398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:35167 (local address 127.0.0.1:37928)
0812 07:08:01.058874 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:01.059006 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":275,"threads_started":1}
W20260812 07:08:01.059692 25279 catalog_manager.cc:4584] TS b3aad6d4634947aca8c854dad3fb9401 (127.6.61.1:35167): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 69062ea8e2714cf8b639da06c2edc1c8: Network error: Client connection negotiation failed: client connection to 127.6.61.1:35167: connect: Connection refused (error 111)
I20260812 07:08:01.081341 25842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:01.058215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42516 (local address 127.6.61.2:40553)
0812 07:08:01.058422 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:01.058429 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:01.058548 (+ 119us) server_negotiation.cc:407] Connection header received
0812 07:08:01.058841 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:01.058847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:01.058941 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:01.059055 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:01.061282 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:01.065459 (+ 4177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:01.067739 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:01.068512 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:01.068677 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:01.072732 (+ 4055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:01.072751 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:01.072774 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:08:01.072829 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:01.077109 (+ 4280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:01.078660 (+ 1551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:01.078672 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:01.078676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:01.078861 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:01.079384 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:01.079390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:01.079393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:01.079864 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:01.080229 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:01.080677 (+ 448us) server_negotiation.cc:299] Negotiation successful
0812 07:08:01.080925 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 07:08:01.082244 25669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:08:01.113799 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:01.113243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:35167 (local address 127.0.0.1:37944)
0812 07:08:01.113488 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:01.113613 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260812 07:08:01.115028 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:01.114599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:35167 (local address 127.0.0.1:37960)
0812 07:08:01.114763 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:01.114864 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260812 07:08:01.175645 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:01.175055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:35167 (local address 127.0.0.1:37962)
0812 07:08:01.175307 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:01.175422 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260812 07:08:01.176784 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:01.176381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:35167 (local address 127.0.0.1:37970)
0812 07:08:01.176532 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:01.176622 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260812 07:08:01.203410 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 25523
I20260812 07:08:01.227499 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 25677
I20260812 07:08:01.242031 25847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:01.228338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45322 (local address 127.6.61.3:46821)
0812 07:08:01.228571 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:01.228576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:01.228594 (+ 18us) server_negotiation.cc:407] Connection header received
0812 07:08:01.228787 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:01.228791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:01.228866 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:01.228950 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:01.230387 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:01.231135 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:01.232536 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:01.232939 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:01.233046 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:01.235878 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:01.235892 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:01.235910 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:08:01.235944 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:01.239119 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:01.239850 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:01.239858 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:01.239860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:01.240007 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:01.240408 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:01.240413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:01.240415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:01.240853 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:01.241110 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:01.241400 (+ 290us) server_negotiation.cc:299] Negotiation successful
0812 07:08:01.241589 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260812 07:08:01.242743 25822 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:08:01.257570 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:01.257058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:35167 (local address 127.0.0.1:37978)
0812 07:08:01.257290 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:01.257391 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260812 07:08:01.258849 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:01.258390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:35167 (local address 127.0.0.1:37980)
0812 07:08:01.258615 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:01.258711 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:35167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260812 07:08:01.334621 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 25677
I20260812 07:08:01.356601 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 25259
I20260812 07:08:01.358052 25333 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:08:01.473881 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 25259
I20260812 07:08:21.501498 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:39783
--webserver_interface=127.6.61.62
--webserver_port=33503
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:46191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:39783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:08:21.904703 25888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:08:21.905030 25888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:08:21.905200 25888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:08:21.912637 25888 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 07:08:21.912766 25888 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 07:08:21.912806 25888 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 07:08:21.912909 25888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:08:21.912997 25888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:08:21.913130 25888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:08:21.913197 25888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:08:21.913271 25888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:08:21.923552 25888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:46191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:39783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:39783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=33503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:08:21.924988 25888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:08:21.926469 25888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:08:21.939450 25893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:21.939539 25897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:21.939493 25895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:08:21.939852 25888 server_base.cc:1034] running on GCE node
I20260812 07:08:21.941114 25888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:08:21.942674 25888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:08:21.944012 25888 hybrid_clock.cc:648] HybridClock initialized: now 1786518501943934 us; error 46 us; skew 500 ppm
I20260812 07:08:21.944438 25888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:08:21 dist-test-slave-69rg krb5kdc[24621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518501, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:08:21.947845 25888 init.cc:377] Logged in from keytab as kudu/127.6.61.62@KRBTEST.COM (short username kudu)
I20260812 07:08:21.949357 25888 webserver.cc:460] Webserver started at http://127.6.61.62:33503/ using document root <none> and password file <none>
I20260812 07:08:21.950145 25888 fs_manager.cc:359] Metadata directory not provided
I20260812 07:08:21.950242 25888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:08:21.955726 25888 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 07:08:21.958740 25907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:21.960232 25888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:08:21.960443 25888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "cf960fc0f93f47f4bbb12e9580b94392"
format_stamp: "Formatted at 2026-08-12 07:07:35 on dist-test-slave-69rg"
server_key: "bce82cc2ad292df4a17751b72e88e4cf"
server_key_iv: "43b5eac11b3a46d174730a9400f0ad99"
server_key_version: "encryptionkey@0"
I20260812 07:08:21.961429 25888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:08:21.986007 25888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:08:21.990622 25888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:08:21.990955 25888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:08:22.011396 25888 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:39783
I20260812 07:08:22.011482 25965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:39783 every 8 connection(s)
I20260812 07:08:22.012951 25888 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:08:22.020095 25966 sys_catalog.cc:263] Verifying existing consensus state
I20260812 07:08:22.020124 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 25888
I20260812 07:08:22.021085 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.1:35167
--local_ip_for_outbound_sockets=127.6.61.1
--tserver_master_addrs=127.6.61.62:39783
--webserver_port=41937
--webserver_interface=127.6.61.1
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:46191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:08:22.023535 25966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392: Bootstrap starting.
I20260812 07:08:22.044744 25966 log.cc:824] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392: Log is configured to *not* fsync() on all Append() calls
I20260812 07:08:22.064416 25966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 07:08:22.065294 25966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392: Bootstrap complete.
I20260812 07:08:22.075654 25966 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf960fc0f93f47f4bbb12e9580b94392" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39783 } }
I20260812 07:08:22.076584 25966 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf960fc0f93f47f4bbb12e9580b94392, State: Initialized, Role: FOLLOWER
I20260812 07:08:22.077286 25966 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf960fc0f93f47f4bbb12e9580b94392" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39783 } }
I20260812 07:08:22.077560 25966 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:08:22.077701 25966 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:08:22.077939 25966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [term 2 FOLLOWER]: Advancing to term 3
I20260812 07:08:22.081768 25966 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf960fc0f93f47f4bbb12e9580b94392" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39783 } }
I20260812 07:08:22.082343 25966 leader_election.cc:302] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf960fc0f93f47f4bbb12e9580b94392; no voters:
I20260812 07:08:22.083055 25966 leader_election.cc:288] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 07:08:22.083210 25970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 07:08:22.084527 25970 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [term 3 LEADER]: Becoming Leader. State: Replica: cf960fc0f93f47f4bbb12e9580b94392, State: Running, Role: LEADER
I20260812 07:08:22.085161 25970 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf960fc0f93f47f4bbb12e9580b94392" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39783 } }
I20260812 07:08:22.085482 25966 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:08:22.088011 25972 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf960fc0f93f47f4bbb12e9580b94392. Latest consensus state: current_term: 3 leader_uuid: "cf960fc0f93f47f4bbb12e9580b94392" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf960fc0f93f47f4bbb12e9580b94392" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39783 } } }
I20260812 07:08:22.088568 25972 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [sys.catalog]: This master's current role is: LEADER
I20260812 07:08:22.088541 25971 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "cf960fc0f93f47f4bbb12e9580b94392" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf960fc0f93f47f4bbb12e9580b94392" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 39783 } } }
I20260812 07:08:22.088918 25971 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392 [sys.catalog]: This master's current role is: LEADER
I20260812 07:08:22.102952 25976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:08:22.111766 25976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:08:22.113356 25976 catalog_manager.cc:1259] Loaded cluster ID: 4f0975c0b8584f9f8832dc83df8f4dd9
I20260812 07:08:22.113515 25976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:08:22.117851 25976 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:08:22.119642 25976 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392: Loaded TSK: 0
I20260812 07:08:22.119865 25976 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cf960fc0f93f47f4bbb12e9580b94392: Loaded TSK: 1
F20260812 07:08:22.121049 25976 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f28aadb89ad google::LogMessage::Fail() at ??:0
@ 0x7f28aadbb88a google::LogMessage::SendToLog() at ??:0
@ 0x7f28aadb84de google::LogMessage::Flush() at ??:0
@ 0x7f28aadb9a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f28aeb1c7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f28aeb0b73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f28c07e71c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f28c07e6783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f28c08722a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f28c08720c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f28c08bc5e9 std::function<>::operator()() at ??:0
@ 0x7f28c07eacdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f28c07e8c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f28c0870610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f28c0870439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f28ada04345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f28ada0791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f28ada07799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f28ad9e2eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f28b0cbc6db start_thread at ??:0
@ 0x7f28a9c5261f clone at ??:0
W20260812 07:08:22.451797 25968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:08:22.452076 25968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:08:22.452231 25968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:08:22.459501 25968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:08:22.459615 25968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:08:22.459818 25968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.1
I20260812 07:08:22.469614 25968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:46191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.1:35167
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.61.1
--webserver_port=41937
--tserver_master_addrs=127.6.61.62:39783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.1
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:08:22.470992 25968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:08:22.472357 25968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:08:22.486778 26018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:22.486809 26019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:22.486824 26021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:08:22.487692 25968 server_base.cc:1034] running on GCE node
I20260812 07:08:22.488476 25968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:08:22.489573 25968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:08:22.490811 25968 hybrid_clock.cc:648] HybridClock initialized: now 1786518502490740 us; error 64 us; skew 500 ppm
I20260812 07:08:22.491221 25968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:08:22 dist-test-slave-69rg krb5kdc[24621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518502, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:08:22.494951 25968 init.cc:377] Logged in from keytab as kudu/127.6.61.1@KRBTEST.COM (short username kudu)
I20260812 07:08:22.496738 25968 webserver.cc:460] Webserver started at http://127.6.61.1:41937/ using document root <none> and password file <none>
I20260812 07:08:22.497509 25968 fs_manager.cc:359] Metadata directory not provided
I20260812 07:08:22.497668 25968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:08:22.503535 25968 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 07:08:22.506484 26028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:22.507896 25968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:08:22.508086 25968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/wal
uuid: "b3aad6d4634947aca8c854dad3fb9401"
format_stamp: "Formatted at 2026-08-12 07:07:36 on dist-test-slave-69rg"
server_key: "1726941850eaa84ad8046bd670747d3b"
server_key_iv: "c522ff3207f99a1056bcea945dc9bb0e"
server_key_version: "encryptionkey@0"
I20260812 07:08:22.509141 25968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:08:22.526867 25968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:08:22.531452 25968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:08:22.531755 25968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:08:22.532811 25968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:08:22.534333 25968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:08:22.534457 25968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:22.534597 25968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:08:22.534695 25968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:22.570864 25968 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.1:35167
I20260812 07:08:22.570955 26159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.1:35167 every 8 connection(s)
I20260812 07:08:22.572374 25968 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
I20260812 07:08:22.575120 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 25968
I20260812 07:08:22.575829 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.2:40553
--local_ip_for_outbound_sockets=127.6.61.2
--tserver_master_addrs=127.6.61.62:39783
--webserver_port=46073
--webserver_interface=127.6.61.2
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:46191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:08:22.578218 26165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:22.575668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.1:60519)
0812 07:08:22.576682 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:22.576933 (+ 251us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":459,"threads_started":1}
W20260812 07:08:22.578998 26163 heartbeater.cc:644] Failed to heartbeat to 127.6.61.62:39783 (0 consecutive failures): Network error: Failed to ping master at 127.6.61.62:39783: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
I20260812 07:08:22.580037 26165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:22.579588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.1:47887)
0812 07:08:22.579791 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:22.579884 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260812 07:08:22.581046 26165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:22.580615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.1:34103)
0812 07:08:22.580787 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:22.580890 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 07:08:22.581698 26163 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 07:08:22.950071 26166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:08:22.950486 26166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:08:22.950664 26166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:08:22.957981 26166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:08:22.958155 26166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:08:22.958359 26166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.2
I20260812 07:08:22.968463 26166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:46191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.2:40553
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.61.2
--webserver_port=46073
--tserver_master_addrs=127.6.61.62:39783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.2
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:08:22.969942 26166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:08:22.971323 26166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:08:22.984994 26171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:22.985035 26172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:22.985008 26174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:08:22.985991 26166 server_base.cc:1034] running on GCE node
I20260812 07:08:22.986662 26166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:08:22.987723 26166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:08:22.988984 26166 hybrid_clock.cc:648] HybridClock initialized: now 1786518502988921 us; error 55 us; skew 500 ppm
I20260812 07:08:22.989382 26166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:08:22 dist-test-slave-69rg krb5kdc[24621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518502, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:08:22.993182 26166 init.cc:377] Logged in from keytab as kudu/127.6.61.2@KRBTEST.COM (short username kudu)
I20260812 07:08:22.994920 26166 webserver.cc:460] Webserver started at http://127.6.61.2:46073/ using document root <none> and password file <none>
I20260812 07:08:22.995658 26166 fs_manager.cc:359] Metadata directory not provided
I20260812 07:08:22.995828 26166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:08:23.001770 26166 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 07:08:23.004791 26181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:23.006484 26166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:08:23.006693 26166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/wal
uuid: "49e9a79960254210b48fe8468c593d01"
format_stamp: "Formatted at 2026-08-12 07:07:36 on dist-test-slave-69rg"
server_key: "e8ebd3b0fcc47f2792330901dea24982"
server_key_iv: "151e547d0e650c7cbc6361fb9a5ef693"
server_key_version: "encryptionkey@0"
I20260812 07:08:23.007745 26166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:08:23.027663 26166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:08:23.032127 26166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:08:23.032434 26166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:08:23.033521 26166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:08:23.035009 26166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:08:23.035143 26166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:23.035295 26166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:08:23.035386 26166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:23.071391 26166 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.2:40553
I20260812 07:08:23.071486 26315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.2:40553 every 8 connection(s)
I20260812 07:08:23.072983 26166 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
I20260812 07:08:23.079385 26318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:23.076447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.2:44595)
0812 07:08:23.077652 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:23.078308 (+ 656us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":399,"threads_started":1}
W20260812 07:08:23.081722 26316 heartbeater.cc:644] Failed to heartbeat to 127.6.61.62:39783 (0 consecutive failures): Network error: Failed to ping master at 127.6.61.62:39783: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
I20260812 07:08:23.083173 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 26166
I20260812 07:08:23.083492 26318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:23.082819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.2:53677)
0812 07:08:23.083136 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:23.083223 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260812 07:08:23.083971 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.3:46821
--local_ip_for_outbound_sockets=127.6.61.3
--tserver_master_addrs=127.6.61.62:39783
--webserver_port=36949
--webserver_interface=127.6.61.3
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:46191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:08:23.084633 26318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:23.084220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.2:39137)
0812 07:08:23.084412 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:23.084507 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260812 07:08:23.084949 26316 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 07:08:23.433549 26319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:08:23.433928 26319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:08:23.434101 26319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:08:23.441310 26319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:08:23.441453 26319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:08:23.441637 26319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.3
I20260812 07:08:23.451217 26319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:46191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.3:46821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.61.3
--webserver_port=36949
--tserver_master_addrs=127.6.61.62:39783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.3
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:08:23.452529 26319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:08:23.453871 26319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:08:23.467491 26325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:23.467425 26327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:23.467468 26329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:08:23.468839 26319 server_base.cc:1034] running on GCE node
I20260812 07:08:23.469568 26319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:08:23.470646 26319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:08:23.471865 26319 hybrid_clock.cc:648] HybridClock initialized: now 1786518503471800 us; error 61 us; skew 500 ppm
I20260812 07:08:23.472168 26319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:08:23 dist-test-slave-69rg krb5kdc[24621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518503, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:08:23.475641 26319 init.cc:377] Logged in from keytab as kudu/127.6.61.3@KRBTEST.COM (short username kudu)
I20260812 07:08:23.477293 26319 webserver.cc:460] Webserver started at http://127.6.61.3:36949/ using document root <none> and password file <none>
I20260812 07:08:23.478127 26319 fs_manager.cc:359] Metadata directory not provided
I20260812 07:08:23.478289 26319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:08:23.484395 26319 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 07:08:23.487648 26336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:23.489038 26319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 07:08:23.489240 26319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/wal
uuid: "9639a9db44224890a8fcaaf1486080f5"
format_stamp: "Formatted at 2026-08-12 07:07:37 on dist-test-slave-69rg"
server_key: "bb48cd9d98f1e2087020ec6f5eac142c"
server_key_iv: "5653395901af7eb1d2e2234c552aed9e"
server_key_version: "encryptionkey@0"
I20260812 07:08:23.490411 26319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:08:23.506135 26319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:08:23.510380 26319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:08:23.510670 26319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:08:23.511806 26319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:08:23.513361 26319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:08:23.513484 26319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:23.513620 26319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:08:23.513713 26319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:23.548858 26319 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.3:46821
I20260812 07:08:23.548918 26462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.3:46821 every 8 connection(s)
I20260812 07:08:23.550438 26319 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
I20260812 07:08:23.555176 26465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:23.553394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.3:56281)
0812 07:08:23.554143 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:23.554347 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":286,"threads_started":1}
W20260812 07:08:23.555895 26463 heartbeater.cc:644] Failed to heartbeat to 127.6.61.62:39783 (0 consecutive failures): Network error: Failed to ping master at 127.6.61.62:39783: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
I20260812 07:08:23.556677 26465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:23.556354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.3:38733)
0812 07:08:23.556501 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:23.556573 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260812 07:08:23.557643 26465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:23.557284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.3:47483)
0812 07:08:23.557443 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:23.557528 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260812 07:08:23.558066 26463 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 07:08:23.558338 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 26319
I20260812 07:08:23.583189 26468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:23.582359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.1:57789)
0812 07:08:23.582856 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:23.582942 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":260,"threads_started":1}
I20260812 07:08:24.086408 26469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:24.085534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.2:43531)
0812 07:08:24.086069 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:24.086157 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":293,"threads_started":1}
I20260812 07:08:24.559510 26470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:24.558639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.3:50279)
0812 07:08:24.559180 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:24.559275 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":297,"threads_started":1}
I20260812 07:08:24.585060 26471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:24.584263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.1:45027)
0812 07:08:24.584784 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:24.584876 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":270,"threads_started":1}
I20260812 07:08:25.088281 26472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:25.087438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.2:60835)
0812 07:08:25.087972 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:25.088085 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":269,"threads_started":1}
I20260812 07:08:25.561264 26474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:25.560476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.3:45945)
0812 07:08:25.560954 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:25.561043 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":234,"threads_started":1}
I20260812 07:08:25.586784 26475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:25.585997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.1:40983)
0812 07:08:25.586536 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:25.586615 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":303,"threads_started":1}
I20260812 07:08:26.090117 26476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:26.089302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.2:35173)
0812 07:08:26.089804 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:26.089936 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":283,"threads_started":1}
I20260812 07:08:26.562846 26477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:26.562138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.3:46049)
0812 07:08:26.562593 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:26.562673 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":242,"threads_started":1}
I20260812 07:08:26.588526 26479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:26.587688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.1:57997)
0812 07:08:26.588226 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:26.588314 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":255,"threads_started":1}
I20260812 07:08:27.091665 26480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:27.090965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.2:50449)
0812 07:08:27.091401 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:27.091480 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":229,"threads_started":1}
I20260812 07:08:27.564675 26481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:27.563795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.3:44509)
0812 07:08:27.564391 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:27.564475 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":253,"threads_started":1}
I20260812 07:08:27.590585 26482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:27.589567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.1:58849)
0812 07:08:27.590185 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:27.590342 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":306,"threads_started":1}
I20260812 07:08:28.093505 26483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:28.092595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.2:33219)
0812 07:08:28.093171 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:28.093286 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":236,"threads_started":1}
I20260812 07:08:28.566951 26484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:28.565921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.3:47063)
0812 07:08:28.566682 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:28.566768 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":485,"threads_started":1}
I20260812 07:08:28.592355 26485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:28.591490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.1:39633)
0812 07:08:28.592076 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:28.592183 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":348,"threads_started":1}
I20260812 07:08:29.095430 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:29.094509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.2:59105)
0812 07:08:29.095110 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:29.095222 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":368,"threads_started":1}
I20260812 07:08:29.568900 26488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:29.568084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.3:35785)
0812 07:08:29.568607 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:29.568723 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":273,"threads_started":1}
I20260812 07:08:29.594558 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:29.593394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.1:48227)
0812 07:08:29.594106 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:29.594203 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":288,"threads_started":1}
I20260812 07:08:30.097591 26490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:30.096506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.2:40437)
0812 07:08:30.097133 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:30.097261 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":327,"threads_started":1}
I20260812 07:08:30.570926 26491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:30.570044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.3:33465)
0812 07:08:30.570594 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:30.570680 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":242,"threads_started":1}
I20260812 07:08:30.596463 26492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:30.595585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.1:60629)
0812 07:08:30.596198 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:30.596281 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":366,"threads_started":1}
I20260812 07:08:31.099628 26493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:31.098677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.2:36543)
0812 07:08:31.099277 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:31.099440 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":344,"threads_started":1}
I20260812 07:08:31.573415 26494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:31.572154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.3:55791)
0812 07:08:31.573023 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:31.573197 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":525,"threads_started":1}
I20260812 07:08:31.598595 26495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:31.597517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.1:59861)
0812 07:08:31.598283 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:31.598388 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":402,"threads_started":1}
I20260812 07:08:32.101631 26496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:32.100767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.2:60099)
0812 07:08:32.101362 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:32.101446 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":305,"threads_started":1}
I20260812 07:08:32.575316 26497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:32.574451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.3:44487)
0812 07:08:32.575019 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:32.575138 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":307,"threads_started":1}
I20260812 07:08:32.600682 26498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:32.599864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.1:38601)
0812 07:08:32.600395 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:32.600482 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":301,"threads_started":1}
I20260812 07:08:33.103505 26499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:33.102739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.2:54439)
0812 07:08:33.103240 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:33.103328 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":271,"threads_started":1}
I20260812 07:08:33.577457 26500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:33.576437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.3:57967)
0812 07:08:33.577087 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:33.577217 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":389,"threads_started":1}
I20260812 07:08:33.602869 26501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:33.601787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.1:50463)
0812 07:08:33.602507 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:33.602657 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":322,"threads_started":1}
I20260812 07:08:34.105440 26502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:34.104549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.2:55109)
0812 07:08:34.105088 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:34.105188 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":278,"threads_started":1}
I20260812 07:08:34.579422 26503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:34.578618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.3:55927)
0812 07:08:34.579115 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:34.579215 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":259,"threads_started":1}
I20260812 07:08:34.605058 26504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:34.604050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.1:47707)
0812 07:08:34.604742 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:34.604874 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":368,"threads_started":1}
I20260812 07:08:35.107388 26505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:35.106583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.2:43659)
0812 07:08:35.107088 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:35.107192 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":252,"threads_started":1}
I20260812 07:08:35.581458 26507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:35.580580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.3:53077)
0812 07:08:35.581157 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:35.581257 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":281,"threads_started":1}
I20260812 07:08:35.606990 26508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:35.606166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.1:41575)
0812 07:08:35.606696 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:35.606787 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":251,"threads_started":1}
I20260812 07:08:36.109591 26509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:36.108613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.2:40573)
0812 07:08:36.109230 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:36.109327 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":326,"threads_started":1}
I20260812 07:08:36.583417 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:36.582588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.3:58747)
0812 07:08:36.583123 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:36.583219 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":256,"threads_started":1}
I20260812 07:08:36.608959 26511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:36.608013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.1:33317)
0812 07:08:36.608607 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:36.608689 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":348,"threads_started":1}
I20260812 07:08:37.111377 26512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:37.110591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.2:59631)
0812 07:08:37.111098 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:37.111177 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":290,"threads_started":1}
I20260812 07:08:37.585275 26514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:37.584459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.3:39627)
0812 07:08:37.584921 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:37.585044 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":237,"threads_started":1}
I20260812 07:08:37.610927 26516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:37.610205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.1:60591)
0812 07:08:37.610650 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:37.610738 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":229,"threads_started":1}
I20260812 07:08:38.113485 26517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:38.112706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.2:44309)
0812 07:08:38.113199 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:38.113291 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":231,"threads_started":1}
I20260812 07:08:38.564532 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 25968
I20260812 07:08:38.585100 26518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:38.565836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51062 (local address 127.6.61.1:35167)
0812 07:08:38.566621 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:38.566647 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:38.566706 (+ 59us) server_negotiation.cc:407] Connection header received
0812 07:08:38.568582 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:38.568602 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:38.569084 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:38.569495 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:38.571368 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:38.572796 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:38.574372 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:38.575165 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:38.575411 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:38.578400 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:38.578408 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:38.578426 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:08:38.578467 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:38.581645 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:38.582515 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:38.582522 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:38.582528 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:38.582716 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:38.583091 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:38.583094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:38.583096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:38.583646 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:38.583886 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:38.584361 (+ 475us) server_negotiation.cc:299] Negotiation successful
0812 07:08:38.584814 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":349,"threads_started":1}
I20260812 07:08:38.586715 26154 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:08:38.587667 26519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:38.586494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.3:60123)
0812 07:08:38.587038 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:38.587204 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":247,"threads_started":1}
I20260812 07:08:38.675241 26521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:38.673893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:39783 (local address 127.6.61.1:55413)
0812 07:08:38.674580 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:38.674820 (+ 240us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.62:39783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":329,"threads_started":1}
I20260812 07:08:38.675290 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 25968
I20260812 07:08:38.700021 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 26166
I20260812 07:08:38.720911 26522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:38.701303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60782 (local address 127.6.61.2:40553)
0812 07:08:38.702008 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:38.702023 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:38.702065 (+ 42us) server_negotiation.cc:407] Connection header received
0812 07:08:38.703763 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:38.703775 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:38.704240 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:38.704572 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:38.706404 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:38.707749 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:38.709230 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:38.710309 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:38.710469 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:38.713482 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:38.713492 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:38.713512 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:08:38.713558 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:38.717005 (+ 3447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:38.717943 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:38.717951 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:38.717959 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:38.718177 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:38.718601 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:38.718604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:38.718606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:38.719244 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:38.719513 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:38.720036 (+ 523us) server_negotiation.cc:299] Negotiation successful
0812 07:08:38.720593 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":324,"threads_started":1}
I20260812 07:08:38.722489 26309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:08:38.809625 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 26166
I20260812 07:08:38.832237 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 26319
I20260812 07:08:38.851733 26524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:38.833484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50562 (local address 127.6.61.3:46821)
0812 07:08:38.834171 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:38.834187 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:38.834232 (+ 45us) server_negotiation.cc:407] Connection header received
0812 07:08:38.835945 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:38.835955 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:38.836405 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:38.836697 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:38.838252 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:38.839527 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:38.840785 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:38.841449 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:38.841585 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:38.844368 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:38.844383 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:38.844406 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:08:38.844459 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:38.847891 (+ 3432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:38.848570 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:38.848577 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:38.848583 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:38.848778 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:38.849148 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:38.849152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:38.849154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:38.850109 (+ 955us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:38.850445 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:38.850943 (+ 498us) server_negotiation.cc:299] Negotiation successful
0812 07:08:38.851400 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":284,"threads_started":1}
I20260812 07:08:38.853227 26457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:08:38.938017 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 26319
2026-08-12T07:08:38Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65380 ms)
[----------] 30 tests from SecurityITest (310219 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:08:39 dist-test-slave-69rg krb5kdc[26528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:08:39 dist-test-slave-69rg krb5kdc[26528](info): set up 2 sockets
Aug 12 07:08:39 dist-test-slave-69rg krb5kdc[26528](info): commencing operation
krb5kdc: starting...
W20260812 07:08:40.198920 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.214s user 0.000s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:08:40 dist-test-slave-69rg krb5kdc[26528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:08:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:08:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
I20260812 07:08:40.544209 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:46557
--webserver_interface=127.6.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:44427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:46557
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:08:40.906203 26544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:08:40.906724 26544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:08:40.906910 26544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:08:40.917394 26544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:08:40.917528 26544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:08:40.917578 26544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:08:40.917639 26544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:08:40.917685 26544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:08:40.928084 26544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:44427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:46557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:46557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:08:40.929493 26544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:08:40.930860 26544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:08:40.944017 26552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:40.944135 26554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:40.944093 26550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:08:40.945182 26544 server_base.cc:1034] running on GCE node
I20260812 07:08:40.946219 26544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:08:40.947909 26544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:08:40.949205 26544 hybrid_clock.cc:648] HybridClock initialized: now 1786518520949127 us; error 71 us; skew 500 ppm
I20260812 07:08:40.949630 26544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:08:40 dist-test-slave-69rg krb5kdc[26528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518520, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:08:40.956969 26544 init.cc:377] Logged in from keytab as kudu/127.6.61.62@KRBTEST.COM (short username kudu)
I20260812 07:08:40.958804 26544 webserver.cc:460] Webserver started at http://127.6.61.62:44001/ using document root <none> and password file <none>
I20260812 07:08:40.959581 26544 fs_manager.cc:359] Metadata directory not provided
I20260812 07:08:40.959739 26544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:08:40.960070 26544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:08:40.963328 26544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "d6a6df3e9f634ceba1679ddeb0637135"
format_stamp: "Formatted at 2026-08-12 07:08:40 on dist-test-slave-69rg"
server_key: "ced8c4488c965d992663480671f6b176"
server_key_iv: "c4c3ca864eddf8cb17dff69de21ebc5c"
server_key_version: "encryptionkey@0"
I20260812 07:08:40.964190 26544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "d6a6df3e9f634ceba1679ddeb0637135"
format_stamp: "Formatted at 2026-08-12 07:08:40 on dist-test-slave-69rg"
server_key: "ced8c4488c965d992663480671f6b176"
server_key_iv: "c4c3ca864eddf8cb17dff69de21ebc5c"
server_key_version: "encryptionkey@0"
I20260812 07:08:40.970116 26544 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260812 07:08:40.973946 26561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:40.975425 26544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:08:40.975662 26544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "d6a6df3e9f634ceba1679ddeb0637135"
format_stamp: "Formatted at 2026-08-12 07:08:40 on dist-test-slave-69rg"
server_key: "ced8c4488c965d992663480671f6b176"
server_key_iv: "c4c3ca864eddf8cb17dff69de21ebc5c"
server_key_version: "encryptionkey@0"
I20260812 07:08:40.975917 26544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:08:41.005292 26544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:08:41.011605 26544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:08:41.011958 26544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:08:41.035480 26544 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:46557
I20260812 07:08:41.035585 26621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:46557 every 8 connection(s)
I20260812 07:08:41.037063 26544 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:08:41.038782 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 26544
I20260812 07:08:41.039095 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:08:41.040220 6388 external_mini_cluster.cc:1428] Setting key e4f2ee62a6bc77b30c49622c5bdc9b5c
I20260812 07:08:41.044487 26622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 07:08:41 dist-test-slave-69rg krb5kdc[26528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:08:41.059463 26622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6a6df3e9f634ceba1679ddeb0637135: Bootstrap starting.
I20260812 07:08:41.064343 26622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6a6df3e9f634ceba1679ddeb0637135: Neither blocks nor log segments found. Creating new log.
I20260812 07:08:41.065851 26622 log.cc:824] T 00000000000000000000000000000000 P d6a6df3e9f634ceba1679ddeb0637135: Log is configured to *not* fsync() on all Append() calls
I20260812 07:08:41.069651 26625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:41.043561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45522 (local address 127.6.61.62:46557)
0812 07:08:41.044569 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:41.044587 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:41.044647 (+ 60us) server_negotiation.cc:407] Connection header received
0812 07:08:41.046900 (+ 2253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:41.046925 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:41.047432 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:41.047840 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:41.049883 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:41.051318 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:41.053220 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:41.054007 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:41.054145 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:41.059448 (+ 5303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:41.059462 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:41.059484 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:08:41.059545 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:41.064007 (+ 4462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:41.064979 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:41.064989 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:41.064999 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:41.065255 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:41.065669 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:41.065674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:41.065678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:41.066621 (+ 943us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:41.067039 (+ 418us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:41.067764 (+ 725us) server_negotiation.cc:299] Negotiation successful
0812 07:08:41.068292 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":386,"threads_started":1}
I20260812 07:08:41.069948 26622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6a6df3e9f634ceba1679ddeb0637135: No bootstrap required, opened a new log
I20260812 07:08:41.078342 26622 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6a6df3e9f634ceba1679ddeb0637135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a6df3e9f634ceba1679ddeb0637135" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 46557 } }
I20260812 07:08:41.078756 26622 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6a6df3e9f634ceba1679ddeb0637135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:08:41.078837 26622 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6a6df3e9f634ceba1679ddeb0637135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a6df3e9f634ceba1679ddeb0637135, State: Initialized, Role: FOLLOWER
I20260812 07:08:41.079425 26622 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6a6df3e9f634ceba1679ddeb0637135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a6df3e9f634ceba1679ddeb0637135" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 46557 } }
I20260812 07:08:41.079646 26622 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6a6df3e9f634ceba1679ddeb0637135 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:08:41.079800 26622 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6a6df3e9f634ceba1679ddeb0637135 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:08:41.079964 26622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6a6df3e9f634ceba1679ddeb0637135 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:08:41.082842 26622 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6a6df3e9f634ceba1679ddeb0637135 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a6df3e9f634ceba1679ddeb0637135" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 46557 } }
I20260812 07:08:41.083333 26622 leader_election.cc:302] T 00000000000000000000000000000000 P d6a6df3e9f634ceba1679ddeb0637135 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6a6df3e9f634ceba1679ddeb0637135; no voters:
I20260812 07:08:41.083964 26622 leader_election.cc:288] T 00000000000000000000000000000000 P d6a6df3e9f634ceba1679ddeb0637135 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:08:41.084133 26627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6a6df3e9f634ceba1679ddeb0637135 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:08:41.084841 26627 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6a6df3e9f634ceba1679ddeb0637135 [term 1 LEADER]: Becoming Leader. State: Replica: d6a6df3e9f634ceba1679ddeb0637135, State: Running, Role: LEADER
I20260812 07:08:41.085418 26627 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6a6df3e9f634ceba1679ddeb0637135 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a6df3e9f634ceba1679ddeb0637135" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 46557 } }
I20260812 07:08:41.086459 26622 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6a6df3e9f634ceba1679ddeb0637135 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:08:41.089182 26629 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6a6df3e9f634ceba1679ddeb0637135 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6a6df3e9f634ceba1679ddeb0637135" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a6df3e9f634ceba1679ddeb0637135" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 46557 } } }
I20260812 07:08:41.089359 26630 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6a6df3e9f634ceba1679ddeb0637135 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6a6df3e9f634ceba1679ddeb0637135. Latest consensus state: current_term: 1 leader_uuid: "d6a6df3e9f634ceba1679ddeb0637135" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a6df3e9f634ceba1679ddeb0637135" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 46557 } } }
I20260812 07:08:41.089691 26629 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6a6df3e9f634ceba1679ddeb0637135 [sys.catalog]: This master's current role is: LEADER
I20260812 07:08:41.089689 26630 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6a6df3e9f634ceba1679ddeb0637135 [sys.catalog]: This master's current role is: LEADER
I20260812 07:08:41.091425 26638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:08:41.097904 26638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:08:41.108409 26638 catalog_manager.cc:1347] Generated new cluster ID: 041d6de40ddf49d5878ebf8962cfa9b3
I20260812 07:08:41.108593 26638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:08:41.119678 26638 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:08:41.121210 26638 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:08:41.130888 26638 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6a6df3e9f634ceba1679ddeb0637135: Generated new TSK 0
I20260812 07:08:41.131898 26638 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
I20260812 07:08:41.222246 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.1:0
--local_ip_for_outbound_sockets=127.6.61.1
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:46557
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:44427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:08:41.606815 26656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:08:41.607198 26656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:08:41.607364 26656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:08:41.614872 26656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:08:41.615048 26656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:08:41.615243 26656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.1
I20260812 07:08:41.625496 26656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:44427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:46557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.1
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:08:41.626977 26656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:08:41.628497 26656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:08:41.642680 26663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:41.642771 26665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:41.642781 26661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:08:41.643842 26656 server_base.cc:1034] running on GCE node
I20260812 07:08:41.644660 26656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:08:41.645720 26656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:08:41.647011 26656 hybrid_clock.cc:648] HybridClock initialized: now 1786518521646956 us; error 43 us; skew 500 ppm
I20260812 07:08:41.647398 26656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:08:41 dist-test-slave-69rg krb5kdc[26528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518521, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:08:41.654505 26656 init.cc:377] Logged in from keytab as kudu/127.6.61.1@KRBTEST.COM (short username kudu)
I20260812 07:08:41.656147 26656 webserver.cc:460] Webserver started at http://127.6.61.1:45237/ using document root <none> and password file <none>
I20260812 07:08:41.656883 26656 fs_manager.cc:359] Metadata directory not provided
I20260812 07:08:41.657034 26656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:08:41.657390 26656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:08:41.660552 26656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-0/data/instance:
uuid: "fd82e4a300fb4c6caea9afdeec0c4147"
format_stamp: "Formatted at 2026-08-12 07:08:41 on dist-test-slave-69rg"
server_key: "74167fc1df7b3f306ece2cf0240467d9"
server_key_iv: "0f817ab52f6da2725542f5c2a5d536cf"
server_key_version: "encryptionkey@0"
I20260812 07:08:41.661381 26656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance:
uuid: "fd82e4a300fb4c6caea9afdeec0c4147"
format_stamp: "Formatted at 2026-08-12 07:08:41 on dist-test-slave-69rg"
server_key: "74167fc1df7b3f306ece2cf0240467d9"
server_key_iv: "0f817ab52f6da2725542f5c2a5d536cf"
server_key_version: "encryptionkey@0"
I20260812 07:08:41.666750 26656 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 07:08:41.670274 26672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:41.671630 26656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260812 07:08:41.671826 26656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-0/wal
uuid: "fd82e4a300fb4c6caea9afdeec0c4147"
format_stamp: "Formatted at 2026-08-12 07:08:41 on dist-test-slave-69rg"
server_key: "74167fc1df7b3f306ece2cf0240467d9"
server_key_iv: "0f817ab52f6da2725542f5c2a5d536cf"
server_key_version: "encryptionkey@0"
I20260812 07:08:41.672050 26656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:08:41.689131 26656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:08:41.693713 26656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:08:41.694084 26656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:08:41.695122 26656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:08:41.696492 26656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:08:41.696595 26656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:41.696756 26656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:08:41.696834 26656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:41.732362 26656 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.1:34001
I20260812 07:08:41.732452 26798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.1:34001 every 8 connection(s)
I20260812 07:08:41.733731 26656 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
I20260812 07:08:41.740790 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 26656
I20260812 07:08:41.741032 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance
I20260812 07:08:41.741963 6388 external_mini_cluster.cc:1428] Setting key 5e3c55ebf551151a44e406da0e2e4df3
Aug 12 07:08:41 dist-test-slave-69rg krb5kdc[26528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518521, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:08:41.759722 26802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:41.736472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:36625 (local address 127.6.61.62:46557)
0812 07:08:41.736947 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:41.736952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:41.737801 (+ 849us) server_negotiation.cc:407] Connection header received
0812 07:08:41.740218 (+ 2417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:41.740225 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:41.740373 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:41.740521 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:41.742890 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:41.744019 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:41.746573 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:41.747075 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:41.747224 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:41.751669 (+ 4445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:41.751678 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:41.751686 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:08:41.751737 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:41.755592 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:41.756658 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:41.756667 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:41.756673 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:41.756851 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:41.757358 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:41.757365 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:41.757371 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:41.757787 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:41.757977 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:41.758969 (+ 992us) server_negotiation.cc:299] Negotiation successful
0812 07:08:41.759262 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":253,"threads_started":1}
I20260812 07:08:41.760782 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:41.736900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:46557 (local address 127.6.61.1:36625)
0812 07:08:41.737616 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:41.737661 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:41.739809 (+ 2148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:41.740938 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:41.740952 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:41.741777 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:08:41.742674 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:41.742708 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:41.744232 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:41.744237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:41.746400 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:41.746419 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:41.747357 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:41.747368 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:41.747751 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:41.748560 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:08:41.748595 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:08:41.751394 (+ 2799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:08:41.755827 (+ 4433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:08:41.755841 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:08:41.755862 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:08:41.756494 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:08:41.756976 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:08:41.756983 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:08:41.756988 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:08:41.757227 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:08:41.757939 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:08:41.757950 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:08:41.758416 (+ 466us) client_negotiation.cc:769] Sending connection context
0812 07:08:41.758719 (+ 303us) client_negotiation.cc:240] Negotiation successful
0812 07:08:41.759362 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":294,"threads_started":1}
I20260812 07:08:41.762527 26799 heartbeater.cc:347] Connected to a master server at 127.6.61.62:46557
I20260812 07:08:41.762929 26799 heartbeater.cc:464] Registering TS with master...
I20260812 07:08:41.763866 26799 heartbeater.cc:511] Master 127.6.61.62:46557 requested a full tablet report, sending...
I20260812 07:08:41.766455 26582 ts_manager.cc:194] Registered new tserver with Master: fd82e4a300fb4c6caea9afdeec0c4147 (127.6.61.1:34001)
I20260812 07:08:41.767946 26582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.1@KRBTEST.COM'} at 127.6.61.1:36625
WARNING: no policy specified for kudu/127.6.61.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
I20260812 07:08:41.821883 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.2:0
--local_ip_for_outbound_sockets=127.6.61.2
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:46557
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:44427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:08:42.181666 26806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:08:42.182093 26806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:08:42.182283 26806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:08:42.190277 26806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:08:42.190399 26806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:08:42.190603 26806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.2
I20260812 07:08:42.201521 26806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:44427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:46557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.2
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:08:42.202972 26806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:08:42.204396 26806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:08:42.218369 26812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:42.218778 26814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:42.218843 26811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:08:42.219408 26806 server_base.cc:1034] running on GCE node
I20260812 07:08:42.220044 26806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:08:42.221086 26806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:08:42.222332 26806 hybrid_clock.cc:648] HybridClock initialized: now 1786518522222258 us; error 63 us; skew 500 ppm
I20260812 07:08:42.222654 26806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:08:42 dist-test-slave-69rg krb5kdc[26528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518522, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:08:42.226306 26806 init.cc:377] Logged in from keytab as kudu/127.6.61.2@KRBTEST.COM (short username kudu)
I20260812 07:08:42.227980 26806 webserver.cc:460] Webserver started at http://127.6.61.2:38453/ using document root <none> and password file <none>
I20260812 07:08:42.229081 26806 fs_manager.cc:359] Metadata directory not provided
I20260812 07:08:42.229220 26806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:08:42.229591 26806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:08:42.233119 26806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-1/data/instance:
uuid: "b7f98ec9cb3540c5aa3add4778180e02"
format_stamp: "Formatted at 2026-08-12 07:08:42 on dist-test-slave-69rg"
server_key: "1081dc636f6388e61531a1136a70ed74"
server_key_iv: "2cd9feabc5ac2d7b9c23b1d548b83896"
server_key_version: "encryptionkey@0"
I20260812 07:08:42.234256 26806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance:
uuid: "b7f98ec9cb3540c5aa3add4778180e02"
format_stamp: "Formatted at 2026-08-12 07:08:42 on dist-test-slave-69rg"
server_key: "1081dc636f6388e61531a1136a70ed74"
server_key_iv: "2cd9feabc5ac2d7b9c23b1d548b83896"
server_key_version: "encryptionkey@0"
I20260812 07:08:42.240624 26806 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 07:08:42.244863 26823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:42.246521 26806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:08:42.246750 26806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-1/wal
uuid: "b7f98ec9cb3540c5aa3add4778180e02"
format_stamp: "Formatted at 2026-08-12 07:08:42 on dist-test-slave-69rg"
server_key: "1081dc636f6388e61531a1136a70ed74"
server_key_iv: "2cd9feabc5ac2d7b9c23b1d548b83896"
server_key_version: "encryptionkey@0"
I20260812 07:08:42.247015 26806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:08:42.264256 26806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:08:42.268862 26806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:08:42.269125 26806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:08:42.270185 26806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:08:42.271577 26806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:08:42.271657 26806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:42.271750 26806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:08:42.271806 26806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:42.307718 26806 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.2:37589
I20260812 07:08:42.307789 26948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.2:37589 every 8 connection(s)
I20260812 07:08:42.309093 26806 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
I20260812 07:08:42.316457 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 26806
I20260812 07:08:42.316682 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance
I20260812 07:08:42.317348 6388 external_mini_cluster.cc:1428] Setting key 3aabf6494549a2cc3f1b8b39405ac75e
Aug 12 07:08:42 dist-test-slave-69rg krb5kdc[26528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518522, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:08:42.332244 26952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:42.311764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:54619 (local address 127.6.61.62:46557)
0812 07:08:42.312288 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:42.312293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:42.313304 (+ 1011us) server_negotiation.cc:407] Connection header received
0812 07:08:42.315489 (+ 2185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:42.315497 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:42.315617 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:42.315760 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:42.318228 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:42.319202 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:42.321178 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:42.321679 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:42.321866 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:42.326005 (+ 4139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:42.326011 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:42.326015 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:08:42.326051 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:42.328897 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:42.329704 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:42.329708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:42.329711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:42.329919 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:42.330391 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:42.330397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:42.330400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:42.330684 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:42.330859 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:42.331577 (+ 718us) server_negotiation.cc:299] Negotiation successful
0812 07:08:42.331822 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":259,"threads_started":1}
I20260812 07:08:42.333642 26954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:42.312172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:46557 (local address 127.6.61.2:54619)
0812 07:08:42.313092 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:42.313149 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:42.315170 (+ 2021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:42.316099 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:42.316113 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:42.317003 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:08:42.317963 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:42.317993 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:42.319394 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:42.319409 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:42.321001 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:42.321016 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:42.321973 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:42.321982 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:42.322258 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:42.323049 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:08:42.323075 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:08:42.325694 (+ 2619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:08:42.329067 (+ 3373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:08:42.329077 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:08:42.329096 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:08:42.329548 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:08:42.330041 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:08:42.330046 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:08:42.330050 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:08:42.330263 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:08:42.330820 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:08:42.330829 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:08:42.331201 (+ 372us) client_negotiation.cc:769] Sending connection context
0812 07:08:42.331484 (+ 283us) client_negotiation.cc:240] Negotiation successful
0812 07:08:42.332082 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":427,"threads_started":1}
I20260812 07:08:42.335348 26949 heartbeater.cc:347] Connected to a master server at 127.6.61.62:46557
I20260812 07:08:42.335697 26949 heartbeater.cc:464] Registering TS with master...
I20260812 07:08:42.336460 26949 heartbeater.cc:511] Master 127.6.61.62:46557 requested a full tablet report, sending...
I20260812 07:08:42.338210 26582 ts_manager.cc:194] Registered new tserver with Master: b7f98ec9cb3540c5aa3add4778180e02 (127.6.61.2:37589)
I20260812 07:08:42.339102 26582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.2@KRBTEST.COM'} at 127.6.61.2:54619
WARNING: no policy specified for kudu/127.6.61.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
I20260812 07:08:42.398595 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.3:0
--local_ip_for_outbound_sockets=127.6.61.3
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:46557
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:44427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:08:42.771776 26799 heartbeater.cc:503] Master 127.6.61.62:46557 was elected leader, sending a full tablet report...
W20260812 07:08:42.772784 26958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:08:42.773118 26958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:08:42.773290 26958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:08:42.781076 26958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:08:42.781231 26958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:08:42.781416 26958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.3
I20260812 07:08:42.791922 26958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:44427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:46557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.3
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:08:42.793357 26958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:08:42.794786 26958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:08:42.808838 26965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:42.808878 26967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:42.808892 26964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:08:42.809782 26958 server_base.cc:1034] running on GCE node
I20260812 07:08:42.810560 26958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:08:42.811640 26958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:08:42.812872 26958 hybrid_clock.cc:648] HybridClock initialized: now 1786518522812827 us; error 29 us; skew 500 ppm
I20260812 07:08:42.813258 26958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:08:42 dist-test-slave-69rg krb5kdc[26528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518522, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:08:42.816860 26958 init.cc:377] Logged in from keytab as kudu/127.6.61.3@KRBTEST.COM (short username kudu)
I20260812 07:08:42.818531 26958 webserver.cc:460] Webserver started at http://127.6.61.3:35379/ using document root <none> and password file <none>
I20260812 07:08:42.819324 26958 fs_manager.cc:359] Metadata directory not provided
I20260812 07:08:42.819492 26958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:08:42.819856 26958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:08:42.823331 26958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-2/data/instance:
uuid: "25f62de56e854bc49c156b4785824a5d"
format_stamp: "Formatted at 2026-08-12 07:08:42 on dist-test-slave-69rg"
server_key: "b7fc2a21801c8b692e2e8637b4c2a586"
server_key_iv: "17d24916142eb9800e3b2c2f2501f0b0"
server_key_version: "encryptionkey@0"
I20260812 07:08:42.824261 26958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance:
uuid: "25f62de56e854bc49c156b4785824a5d"
format_stamp: "Formatted at 2026-08-12 07:08:42 on dist-test-slave-69rg"
server_key: "b7fc2a21801c8b692e2e8637b4c2a586"
server_key_iv: "17d24916142eb9800e3b2c2f2501f0b0"
server_key_version: "encryptionkey@0"
I20260812 07:08:42.830570 26958 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 07:08:42.835382 26975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:42.836874 26958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260812 07:08:42.837092 26958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-2/wal
uuid: "25f62de56e854bc49c156b4785824a5d"
format_stamp: "Formatted at 2026-08-12 07:08:42 on dist-test-slave-69rg"
server_key: "b7fc2a21801c8b692e2e8637b4c2a586"
server_key_iv: "17d24916142eb9800e3b2c2f2501f0b0"
server_key_version: "encryptionkey@0"
I20260812 07:08:42.837339 26958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:08:42.860769 26958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:08:42.865761 26958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:08:42.866139 26958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:08:42.867295 26958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:08:42.868882 26958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:08:42.869007 26958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:42.869190 26958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:08:42.869277 26958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:42.910115 26958 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.3:42583
I20260812 07:08:42.910213 27097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.3:42583 every 8 connection(s)
I20260812 07:08:42.911509 26958 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
I20260812 07:08:42.914680 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 26958
I20260812 07:08:42.914907 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance
I20260812 07:08:42.915560 6388 external_mini_cluster.cc:1428] Setting key 9dd6000baa36a1430404ac1d9ee88fac
Aug 12 07:08:42 dist-test-slave-69rg krb5kdc[26528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518522, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:08:42.934957 27100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:42.914740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:59837 (local address 127.6.61.62:46557)
0812 07:08:42.915313 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:42.915318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:42.916507 (+ 1189us) server_negotiation.cc:407] Connection header received
0812 07:08:42.919188 (+ 2681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:42.919194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:42.919309 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:42.919449 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:42.921100 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:42.922014 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:42.923635 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:42.924054 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:42.924201 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:42.928023 (+ 3822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:42.928028 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:42.928032 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:08:42.928068 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:42.931498 (+ 3430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:42.932567 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:42.932572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:42.932575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:42.932722 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:42.933164 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:42.933168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:42.933170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:42.933477 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:42.933601 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:42.934462 (+ 861us) server_negotiation.cc:299] Negotiation successful
0812 07:08:42.934653 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":309,"threads_started":1}
I20260812 07:08:42.935971 27101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:42.915381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:46557 (local address 127.6.61.3:59837)
0812 07:08:42.916237 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:42.916341 (+ 104us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:42.918719 (+ 2378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:42.919720 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:42.919729 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:42.920297 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:08:42.920917 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:42.920935 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:42.922181 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:42.922185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:42.923467 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:42.923480 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:42.924305 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:42.924314 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:42.924559 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:42.925284 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:08:42.925314 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:08:42.927799 (+ 2485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:08:42.931658 (+ 3859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:08:42.931668 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:08:42.931687 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:08:42.932418 (+ 731us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:08:42.932853 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:08:42.932856 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:08:42.932858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:08:42.933049 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:08:42.933627 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:08:42.933634 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:08:42.934077 (+ 443us) client_negotiation.cc:769] Sending connection context
0812 07:08:42.934357 (+ 280us) client_negotiation.cc:240] Negotiation successful
0812 07:08:42.934788 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":362,"threads_started":1}
I20260812 07:08:42.937427 27098 heartbeater.cc:347] Connected to a master server at 127.6.61.62:46557
I20260812 07:08:42.937757 27098 heartbeater.cc:464] Registering TS with master...
I20260812 07:08:42.938583 27098 heartbeater.cc:511] Master 127.6.61.62:46557 requested a full tablet report, sending...
I20260812 07:08:42.940078 26582 ts_manager.cc:194] Registered new tserver with Master: 25f62de56e854bc49c156b4785824a5d (127.6.61.3:42583)
I20260812 07:08:42.941038 26582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.3@KRBTEST.COM'} at 127.6.61.3:59837
I20260812 07:08:42.944659 6388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 07:08:42.963801 27100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:42.949479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45536 (local address 127.6.61.62:46557)
0812 07:08:42.949732 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:42.949739 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:42.950185 (+ 446us) server_negotiation.cc:407] Connection header received
0812 07:08:42.950346 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:42.950350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:42.950417 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:42.950505 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:42.952121 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:42.953017 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:42.954588 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:42.955006 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:42.955127 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:42.957664 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:42.957669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:42.957673 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:08:42.957720 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:42.960825 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:42.961579 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:42.961583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:42.961586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:42.961724 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:42.962164 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:42.962169 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:42.962171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:42.962549 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:42.962755 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:42.963124 (+ 369us) server_negotiation.cc:299] Negotiation successful
0812 07:08:42.963453 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 07:08:42.968083 26582 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:08:42.975540 26582 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:08:43.000200 27115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:42.989100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:37589 (local address 127.0.0.1:34866)
0812 07:08:42.990243 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:42.990281 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:42.990521 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:42.991685 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:42.991696 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:42.991758 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:08:42.992271 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:42.992298 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:42.994657 (+ 2359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:42.994663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:42.996880 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:42.996900 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:42.998925 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:42.998933 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:42.999223 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:42.999266 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 07:08:42.999427 (+ 161us) client_negotiation.cc:240] Negotiation successful
0812 07:08:42.999660 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":896,"mutex_wait_us":1,"thread_start_us":739,"threads_started":1}
I20260812 07:08:43.000468 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:42.990172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42914 (local address 127.6.61.3:42583)
0812 07:08:42.990991 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:42.991004 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:42.991760 (+ 756us) server_negotiation.cc:407] Connection header received
0812 07:08:42.991854 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:42.991858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:42.992051 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:42.992246 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:08:42.993307 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:42.995076 (+ 1769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:42.997561 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:42.999261 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:42.999398 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:42.999610 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:42.999907 (+ 297us) server_negotiation.cc:299] Negotiation successful
0812 07:08:43.000103 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":592,"threads_started":1}
I20260812 07:08:43.000490 27116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:42.989157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:42583 (local address 127.0.0.1:42914)
0812 07:08:42.991457 (+ 2300us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:42.991475 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:42.991611 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:42.992535 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:42.992543 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:42.992574 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:08:42.993055 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:42.993068 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:42.995236 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:42.995242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:42.997327 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:42.997345 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:42.999498 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:42.999504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:42.999693 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:42.999725 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 07:08:42.999829 (+ 104us) client_negotiation.cc:240] Negotiation successful
0812 07:08:42.999963 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2125,"thread_start_us":2187,"threads_started":1}
I20260812 07:08:43.001305 27114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:42.989082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34866 (local address 127.6.61.2:37589)
0812 07:08:42.989757 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:42.989769 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:42.990410 (+ 641us) server_negotiation.cc:407] Connection header received
0812 07:08:42.990796 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:42.990806 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:42.991077 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:42.991330 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:08:42.992527 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:42.994470 (+ 1943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:42.997076 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:42.998647 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:42.998801 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:42.998999 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:43.000743 (+ 1744us) server_negotiation.cc:299] Negotiation successful
0812 07:08:43.000941 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":359,"threads_started":1}
I20260812 07:08:43.001919 27118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:42.991823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:34001 (local address 127.0.0.1:53860)
0812 07:08:42.992490 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:42.992509 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:42.992610 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:42.993703 (+ 1093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:42.993708 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:42.993733 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:08:42.994147 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:42.994160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:42.996580 (+ 2420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:42.996587 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:42.999000 (+ 2413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:42.999014 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:43.001046 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:43.001052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:43.001240 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:43.001280 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 07:08:43.001377 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 07:08:43.001523 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":43,"thread_start_us":924,"threads_started":1}
I20260812 07:08:43.002143 27119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:42.992529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53860 (local address 127.6.61.1:34001)
0812 07:08:42.993264 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:42.993270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:42.993292 (+ 22us) server_negotiation.cc:407] Connection header received
0812 07:08:42.993363 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:42.993368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:42.993516 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:42.993702 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:08:42.994850 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:42.996418 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:42.999202 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:43.000856 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:43.001004 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:43.001222 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:43.001589 (+ 367us) server_negotiation.cc:299] Negotiation successful
0812 07:08:43.001769 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":378,"threads_started":1}
I20260812 07:08:43.005630 26723 tablet_service.cc:1467] Processing CreateTablet for tablet 1df0669b69ac461d9f77a40be73312f2 (DEFAULT_TABLE table=test-table [id=8d037b3a1cab442ca8521e34b645f238]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:08:43.005743 26877 tablet_service.cc:1467] Processing CreateTablet for tablet 1df0669b69ac461d9f77a40be73312f2 (DEFAULT_TABLE table=test-table [id=8d037b3a1cab442ca8521e34b645f238]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:08:43.005752 27028 tablet_service.cc:1467] Processing CreateTablet for tablet 1df0669b69ac461d9f77a40be73312f2 (DEFAULT_TABLE table=test-table [id=8d037b3a1cab442ca8521e34b645f238]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:08:43.007205 27028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1df0669b69ac461d9f77a40be73312f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:08:43.007215 26877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1df0669b69ac461d9f77a40be73312f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:08:43.007263 26723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1df0669b69ac461d9f77a40be73312f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:08:43.019769 27120 tablet_bootstrap.cc:492] T 1df0669b69ac461d9f77a40be73312f2 P fd82e4a300fb4c6caea9afdeec0c4147: Bootstrap starting.
I20260812 07:08:43.022462 27121 tablet_bootstrap.cc:492] T 1df0669b69ac461d9f77a40be73312f2 P b7f98ec9cb3540c5aa3add4778180e02: Bootstrap starting.
I20260812 07:08:43.024202 27122 tablet_bootstrap.cc:492] T 1df0669b69ac461d9f77a40be73312f2 P 25f62de56e854bc49c156b4785824a5d: Bootstrap starting.
I20260812 07:08:43.025238 27120 tablet_bootstrap.cc:654] T 1df0669b69ac461d9f77a40be73312f2 P fd82e4a300fb4c6caea9afdeec0c4147: Neither blocks nor log segments found. Creating new log.
I20260812 07:08:43.026512 27121 tablet_bootstrap.cc:654] T 1df0669b69ac461d9f77a40be73312f2 P b7f98ec9cb3540c5aa3add4778180e02: Neither blocks nor log segments found. Creating new log.
I20260812 07:08:43.026774 27120 log.cc:824] T 1df0669b69ac461d9f77a40be73312f2 P fd82e4a300fb4c6caea9afdeec0c4147: Log is configured to *not* fsync() on all Append() calls
I20260812 07:08:43.027231 27122 tablet_bootstrap.cc:654] T 1df0669b69ac461d9f77a40be73312f2 P 25f62de56e854bc49c156b4785824a5d: Neither blocks nor log segments found. Creating new log.
I20260812 07:08:43.028000 27121 log.cc:824] T 1df0669b69ac461d9f77a40be73312f2 P b7f98ec9cb3540c5aa3add4778180e02: Log is configured to *not* fsync() on all Append() calls
I20260812 07:08:43.028204 27122 log.cc:824] T 1df0669b69ac461d9f77a40be73312f2 P 25f62de56e854bc49c156b4785824a5d: Log is configured to *not* fsync() on all Append() calls
I20260812 07:08:43.031952 27122 tablet_bootstrap.cc:492] T 1df0669b69ac461d9f77a40be73312f2 P 25f62de56e854bc49c156b4785824a5d: No bootstrap required, opened a new log
I20260812 07:08:43.032281 27121 tablet_bootstrap.cc:492] T 1df0669b69ac461d9f77a40be73312f2 P b7f98ec9cb3540c5aa3add4778180e02: No bootstrap required, opened a new log
I20260812 07:08:43.032298 27122 ts_tablet_manager.cc:1397] T 1df0669b69ac461d9f77a40be73312f2 P 25f62de56e854bc49c156b4785824a5d: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260812 07:08:43.032603 27121 ts_tablet_manager.cc:1397] T 1df0669b69ac461d9f77a40be73312f2 P b7f98ec9cb3540c5aa3add4778180e02: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260812 07:08:43.035072 27120 tablet_bootstrap.cc:492] T 1df0669b69ac461d9f77a40be73312f2 P fd82e4a300fb4c6caea9afdeec0c4147: No bootstrap required, opened a new log
I20260812 07:08:43.035459 27120 ts_tablet_manager.cc:1397] T 1df0669b69ac461d9f77a40be73312f2 P fd82e4a300fb4c6caea9afdeec0c4147: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.002s
I20260812 07:08:43.042979 27122 raft_consensus.cc:348] T 1df0669b69ac461d9f77a40be73312f2 P 25f62de56e854bc49c156b4785824a5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f98ec9cb3540c5aa3add4778180e02" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 37589 } } peers { permanent_uuid: "25f62de56e854bc49c156b4785824a5d" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 42583 } } peers { permanent_uuid: "fd82e4a300fb4c6caea9afdeec0c4147" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34001 } }
I20260812 07:08:43.043576 27122 raft_consensus.cc:374] T 1df0669b69ac461d9f77a40be73312f2 P 25f62de56e854bc49c156b4785824a5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:08:43.043720 27122 raft_consensus.cc:729] T 1df0669b69ac461d9f77a40be73312f2 P 25f62de56e854bc49c156b4785824a5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25f62de56e854bc49c156b4785824a5d, State: Initialized, Role: FOLLOWER
I20260812 07:08:43.044462 27122 consensus_queue.cc:260] T 1df0669b69ac461d9f77a40be73312f2 P 25f62de56e854bc49c156b4785824a5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f98ec9cb3540c5aa3add4778180e02" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 37589 } } peers { permanent_uuid: "25f62de56e854bc49c156b4785824a5d" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 42583 } } peers { permanent_uuid: "fd82e4a300fb4c6caea9afdeec0c4147" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34001 } }
I20260812 07:08:43.044996 27121 raft_consensus.cc:348] T 1df0669b69ac461d9f77a40be73312f2 P b7f98ec9cb3540c5aa3add4778180e02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f98ec9cb3540c5aa3add4778180e02" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 37589 } } peers { permanent_uuid: "25f62de56e854bc49c156b4785824a5d" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 42583 } } peers { permanent_uuid: "fd82e4a300fb4c6caea9afdeec0c4147" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34001 } }
I20260812 07:08:43.045588 27121 raft_consensus.cc:374] T 1df0669b69ac461d9f77a40be73312f2 P b7f98ec9cb3540c5aa3add4778180e02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:08:43.045802 27121 raft_consensus.cc:729] T 1df0669b69ac461d9f77a40be73312f2 P b7f98ec9cb3540c5aa3add4778180e02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7f98ec9cb3540c5aa3add4778180e02, State: Initialized, Role: FOLLOWER
I20260812 07:08:43.046690 27121 consensus_queue.cc:260] T 1df0669b69ac461d9f77a40be73312f2 P b7f98ec9cb3540c5aa3add4778180e02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f98ec9cb3540c5aa3add4778180e02" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 37589 } } peers { permanent_uuid: "25f62de56e854bc49c156b4785824a5d" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 42583 } } peers { permanent_uuid: "fd82e4a300fb4c6caea9afdeec0c4147" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34001 } }
I20260812 07:08:43.050846 27120 raft_consensus.cc:348] T 1df0669b69ac461d9f77a40be73312f2 P fd82e4a300fb4c6caea9afdeec0c4147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f98ec9cb3540c5aa3add4778180e02" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 37589 } } peers { permanent_uuid: "25f62de56e854bc49c156b4785824a5d" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 42583 } } peers { permanent_uuid: "fd82e4a300fb4c6caea9afdeec0c4147" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34001 } }
I20260812 07:08:43.051817 27120 raft_consensus.cc:374] T 1df0669b69ac461d9f77a40be73312f2 P fd82e4a300fb4c6caea9afdeec0c4147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:08:43.052042 27120 raft_consensus.cc:729] T 1df0669b69ac461d9f77a40be73312f2 P fd82e4a300fb4c6caea9afdeec0c4147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd82e4a300fb4c6caea9afdeec0c4147, State: Initialized, Role: FOLLOWER
I20260812 07:08:43.055299 27120 consensus_queue.cc:260] T 1df0669b69ac461d9f77a40be73312f2 P fd82e4a300fb4c6caea9afdeec0c4147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f98ec9cb3540c5aa3add4778180e02" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 37589 } } peers { permanent_uuid: "25f62de56e854bc49c156b4785824a5d" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 42583 } } peers { permanent_uuid: "fd82e4a300fb4c6caea9afdeec0c4147" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34001 } }
I20260812 07:08:43.055595 27121 ts_tablet_manager.cc:1428] T 1df0669b69ac461d9f77a40be73312f2 P b7f98ec9cb3540c5aa3add4778180e02: Time spent starting tablet: real 0.023s user 0.016s sys 0.004s
I20260812 07:08:43.056097 26949 heartbeater.cc:503] Master 127.6.61.62:46557 was elected leader, sending a full tablet report...
I20260812 07:08:43.059944 27098 heartbeater.cc:503] Master 127.6.61.62:46557 was elected leader, sending a full tablet report...
I20260812 07:08:43.060698 27122 ts_tablet_manager.cc:1428] T 1df0669b69ac461d9f77a40be73312f2 P 25f62de56e854bc49c156b4785824a5d: Time spent starting tablet: real 0.028s user 0.020s sys 0.008s
W20260812 07:08:43.061465 26950 tablet.cc:2369] T 1df0669b69ac461d9f77a40be73312f2 P b7f98ec9cb3540c5aa3add4778180e02: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:08:43.063796 27120 ts_tablet_manager.cc:1428] T 1df0669b69ac461d9f77a40be73312f2 P fd82e4a300fb4c6caea9afdeec0c4147: Time spent starting tablet: real 0.028s user 0.013s sys 0.009s
I20260812 07:08:43.093009 27127 raft_consensus.cc:482] T 1df0669b69ac461d9f77a40be73312f2 P b7f98ec9cb3540c5aa3add4778180e02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:08:43.093343 27127 raft_consensus.cc:504] T 1df0669b69ac461d9f77a40be73312f2 P b7f98ec9cb3540c5aa3add4778180e02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f98ec9cb3540c5aa3add4778180e02" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 37589 } } peers { permanent_uuid: "25f62de56e854bc49c156b4785824a5d" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 42583 } } peers { permanent_uuid: "fd82e4a300fb4c6caea9afdeec0c4147" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34001 } }
I20260812 07:08:43.095068 27127 leader_election.cc:288] T 1df0669b69ac461d9f77a40be73312f2 P b7f98ec9cb3540c5aa3add4778180e02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25f62de56e854bc49c156b4785824a5d (127.6.61.3:42583), fd82e4a300fb4c6caea9afdeec0c4147 (127.6.61.1:34001)
I20260812 07:08:43.101943 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:43.095547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:45405 (local address 127.6.61.3:42583)
0812 07:08:43.095760 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:43.095765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:43.096313 (+ 548us) server_negotiation.cc:407] Connection header received
0812 07:08:43.096624 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:43.096629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:43.096716 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:43.096851 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:08:43.097436 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:43.098291 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:43.100104 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:43.100905 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:43.101063 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:43.101184 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:43.101497 (+ 313us) server_negotiation.cc:299] Negotiation successful
0812 07:08:43.101630 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 07:08:43.102058 27129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:43.095452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:42583 (local address 127.6.61.2:45405)
0812 07:08:43.096194 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:43.096217 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:43.096371 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:43.096879 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:43.096885 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:43.096929 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:08:43.097275 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:43.097285 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:43.098430 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:43.098437 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:43.099927 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:43.099941 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:43.101132 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:43.101138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:43.101320 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:43.101346 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 07:08:43.101464 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 07:08:43.101607 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":394,"threads_started":1}
I20260812 07:08:43.102592 27119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:43.096567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:60613 (local address 127.6.61.1:34001)
0812 07:08:43.097107 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:43.097114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:43.097194 (+ 80us) server_negotiation.cc:407] Connection header received
0812 07:08:43.097408 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:43.097412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:43.097472 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:43.097581 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:08:43.098235 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:43.098936 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:43.100701 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:43.101506 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:43.101669 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:43.101787 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:43.102201 (+ 414us) server_negotiation.cc:299] Negotiation successful
0812 07:08:43.102336 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383}
I20260812 07:08:43.103406 27130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:43.096358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:34001 (local address 127.6.61.2:60613)
0812 07:08:43.097087 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:43.097105 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:43.097260 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:43.097604 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:43.097608 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:43.097634 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:08:43.098059 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:43.098069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:43.099093 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:43.099098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:43.100527 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:43.100540 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:43.101757 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:43.101764 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:43.101972 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:43.101995 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 07:08:43.102906 (+ 911us) client_negotiation.cc:240] Negotiation successful
0812 07:08:43.103063 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":380,"threads_started":1}
I20260812 07:08:43.103516 27049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df0669b69ac461d9f77a40be73312f2" candidate_uuid: "b7f98ec9cb3540c5aa3add4778180e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25f62de56e854bc49c156b4785824a5d" is_pre_election: true
I20260812 07:08:43.104127 27049 raft_consensus.cc:2454] T 1df0669b69ac461d9f77a40be73312f2 P 25f62de56e854bc49c156b4785824a5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7f98ec9cb3540c5aa3add4778180e02 in term 0.
I20260812 07:08:43.104558 26749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df0669b69ac461d9f77a40be73312f2" candidate_uuid: "b7f98ec9cb3540c5aa3add4778180e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd82e4a300fb4c6caea9afdeec0c4147" is_pre_election: true
I20260812 07:08:43.105129 26749 raft_consensus.cc:2454] T 1df0669b69ac461d9f77a40be73312f2 P fd82e4a300fb4c6caea9afdeec0c4147 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7f98ec9cb3540c5aa3add4778180e02 in term 0.
I20260812 07:08:43.105152 26826 leader_election.cc:302] T 1df0669b69ac461d9f77a40be73312f2 P b7f98ec9cb3540c5aa3add4778180e02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25f62de56e854bc49c156b4785824a5d, b7f98ec9cb3540c5aa3add4778180e02; no voters:
I20260812 07:08:43.105629 27127 raft_consensus.cc:2781] T 1df0669b69ac461d9f77a40be73312f2 P b7f98ec9cb3540c5aa3add4778180e02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:08:43.105792 27127 raft_consensus.cc:482] T 1df0669b69ac461d9f77a40be73312f2 P b7f98ec9cb3540c5aa3add4778180e02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:08:43.105937 27127 raft_consensus.cc:3037] T 1df0669b69ac461d9f77a40be73312f2 P b7f98ec9cb3540c5aa3add4778180e02 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:08:43.108203 27127 raft_consensus.cc:504] T 1df0669b69ac461d9f77a40be73312f2 P b7f98ec9cb3540c5aa3add4778180e02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f98ec9cb3540c5aa3add4778180e02" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 37589 } } peers { permanent_uuid: "25f62de56e854bc49c156b4785824a5d" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 42583 } } peers { permanent_uuid: "fd82e4a300fb4c6caea9afdeec0c4147" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34001 } }
I20260812 07:08:43.108837 27127 leader_election.cc:288] T 1df0669b69ac461d9f77a40be73312f2 P b7f98ec9cb3540c5aa3add4778180e02 [CANDIDATE]: Term 1 election: Requested vote from peers 25f62de56e854bc49c156b4785824a5d (127.6.61.3:42583), fd82e4a300fb4c6caea9afdeec0c4147 (127.6.61.1:34001)
I20260812 07:08:43.109355 27049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df0669b69ac461d9f77a40be73312f2" candidate_uuid: "b7f98ec9cb3540c5aa3add4778180e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25f62de56e854bc49c156b4785824a5d"
I20260812 07:08:43.109411 26749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df0669b69ac461d9f77a40be73312f2" candidate_uuid: "b7f98ec9cb3540c5aa3add4778180e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd82e4a300fb4c6caea9afdeec0c4147"
I20260812 07:08:43.109565 27049 raft_consensus.cc:3037] T 1df0669b69ac461d9f77a40be73312f2 P 25f62de56e854bc49c156b4785824a5d [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:08:43.109688 26749 raft_consensus.cc:3037] T 1df0669b69ac461d9f77a40be73312f2 P fd82e4a300fb4c6caea9afdeec0c4147 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:08:43.111913 27049 raft_consensus.cc:2454] T 1df0669b69ac461d9f77a40be73312f2 P 25f62de56e854bc49c156b4785824a5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7f98ec9cb3540c5aa3add4778180e02 in term 1.
I20260812 07:08:43.112022 26749 raft_consensus.cc:2454] T 1df0669b69ac461d9f77a40be73312f2 P fd82e4a300fb4c6caea9afdeec0c4147 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7f98ec9cb3540c5aa3add4778180e02 in term 1.
I20260812 07:08:43.112329 26826 leader_election.cc:302] T 1df0669b69ac461d9f77a40be73312f2 P b7f98ec9cb3540c5aa3add4778180e02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25f62de56e854bc49c156b4785824a5d, b7f98ec9cb3540c5aa3add4778180e02; no voters:
I20260812 07:08:43.112686 27127 raft_consensus.cc:2781] T 1df0669b69ac461d9f77a40be73312f2 P b7f98ec9cb3540c5aa3add4778180e02 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:08:43.113117 27127 raft_consensus.cc:686] T 1df0669b69ac461d9f77a40be73312f2 P b7f98ec9cb3540c5aa3add4778180e02 [term 1 LEADER]: Becoming Leader. State: Replica: b7f98ec9cb3540c5aa3add4778180e02, State: Running, Role: LEADER
I20260812 07:08:43.113626 27127 consensus_queue.cc:237] T 1df0669b69ac461d9f77a40be73312f2 P b7f98ec9cb3540c5aa3add4778180e02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f98ec9cb3540c5aa3add4778180e02" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 37589 } } peers { permanent_uuid: "25f62de56e854bc49c156b4785824a5d" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 42583 } } peers { permanent_uuid: "fd82e4a300fb4c6caea9afdeec0c4147" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34001 } }
I20260812 07:08:43.118752 26582 catalog_manager.cc:5495] T 1df0669b69ac461d9f77a40be73312f2 P b7f98ec9cb3540c5aa3add4778180e02 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7f98ec9cb3540c5aa3add4778180e02 (127.6.61.2). New cstate: current_term: 1 leader_uuid: "b7f98ec9cb3540c5aa3add4778180e02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f98ec9cb3540c5aa3add4778180e02" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 37589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25f62de56e854bc49c156b4785824a5d" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 42583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd82e4a300fb4c6caea9afdeec0c4147" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34001 } health_report { overall_health: UNKNOWN } } }
I20260812 07:08:43.160249 27114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:43.154038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34868 (local address 127.6.61.2:37589)
0812 07:08:43.154266 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:43.154273 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:43.154386 (+ 113us) server_negotiation.cc:407] Connection header received
0812 07:08:43.154665 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:43.154669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:43.154771 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:43.154945 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:08:43.155396 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:43.156141 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:43.157924 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:43.158388 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:43.158539 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:43.159059 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:08:43.159605 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:08:43.159790 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:43.159845 (+ 55us) server_negotiation.cc:299] Negotiation successful
0812 07:08:43.159932 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260812 07:08:43.163331 27099 tablet.cc:2369] T 1df0669b69ac461d9f77a40be73312f2 P 25f62de56e854bc49c156b4785824a5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:08:43.169348 27049 raft_consensus.cc:1261] T 1df0669b69ac461d9f77a40be73312f2 P 25f62de56e854bc49c156b4785824a5d [term 1 FOLLOWER]: Refusing update from remote peer b7f98ec9cb3540c5aa3add4778180e02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:08:43.169411 26749 raft_consensus.cc:1261] T 1df0669b69ac461d9f77a40be73312f2 P fd82e4a300fb4c6caea9afdeec0c4147 [term 1 FOLLOWER]: Refusing update from remote peer b7f98ec9cb3540c5aa3add4778180e02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:08:43.170468 27127 consensus_queue.cc:1035] T 1df0669b69ac461d9f77a40be73312f2 P b7f98ec9cb3540c5aa3add4778180e02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25f62de56e854bc49c156b4785824a5d" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 42583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:08:43.170836 27132 consensus_queue.cc:1035] T 1df0669b69ac461d9f77a40be73312f2 P b7f98ec9cb3540c5aa3add4778180e02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd82e4a300fb4c6caea9afdeec0c4147" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 34001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:08:43.184183 27138 mvcc.cc:205] Tried to move back new op lower bound from 7317579870891286528 to 7317579870677295104. Current Snapshot: MvccSnapshot[applied={T|T < 7317579870891286528}]
I20260812 07:08:43.210405 26582 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:08:43.210670 26582 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:08:43.214365 26582 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6a6df3e9f634ceba1679ddeb0637135: Sending DeleteTablet for 3 replicas of tablet 1df0669b69ac461d9f77a40be73312f2
I20260812 07:08:43.215660 26877 tablet_service.cc:1514] Processing DeleteTablet for tablet 1df0669b69ac461d9f77a40be73312f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:08:43 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:34866
I20260812 07:08:43.215889 27028 tablet_service.cc:1514] Processing DeleteTablet for tablet 1df0669b69ac461d9f77a40be73312f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:08:43 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:42914
I20260812 07:08:43.216305 26723 tablet_service.cc:1514] Processing DeleteTablet for tablet 1df0669b69ac461d9f77a40be73312f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:08:43 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:53860
I20260812 07:08:43.217517 27153 tablet_replica.cc:330] T 1df0669b69ac461d9f77a40be73312f2 P fd82e4a300fb4c6caea9afdeec0c4147: stopping tablet replica
I20260812 07:08:43.217670 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 26656
I20260812 07:08:43.218125 27153 raft_consensus.cc:2229] T 1df0669b69ac461d9f77a40be73312f2 P fd82e4a300fb4c6caea9afdeec0c4147 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:08:43.218789 27153 raft_consensus.cc:2258] T 1df0669b69ac461d9f77a40be73312f2 P fd82e4a300fb4c6caea9afdeec0c4147 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:08:43.219725 27152 tablet_replica.cc:330] T 1df0669b69ac461d9f77a40be73312f2 P 25f62de56e854bc49c156b4785824a5d: stopping tablet replica
I20260812 07:08:43.220175 27152 raft_consensus.cc:2229] T 1df0669b69ac461d9f77a40be73312f2 P 25f62de56e854bc49c156b4785824a5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:08:43.220759 27152 raft_consensus.cc:2258] T 1df0669b69ac461d9f77a40be73312f2 P 25f62de56e854bc49c156b4785824a5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:08:43.221657 27151 tablet_replica.cc:330] T 1df0669b69ac461d9f77a40be73312f2 P b7f98ec9cb3540c5aa3add4778180e02: stopping tablet replica
I20260812 07:08:43.222101 27151 raft_consensus.cc:2229] T 1df0669b69ac461d9f77a40be73312f2 P b7f98ec9cb3540c5aa3add4778180e02 [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:08:43.222780 27151 raft_consensus.cc:2258] T 1df0669b69ac461d9f77a40be73312f2 P b7f98ec9cb3540c5aa3add4778180e02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:08:43.223042 27152 ts_tablet_manager.cc:1905] T 1df0669b69ac461d9f77a40be73312f2 P 25f62de56e854bc49c156b4785824a5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:08:43.224995 27151 ts_tablet_manager.cc:1905] T 1df0669b69ac461d9f77a40be73312f2 P b7f98ec9cb3540c5aa3add4778180e02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:08:43.225390 27153 ts_tablet_manager.cc:1905] T 1df0669b69ac461d9f77a40be73312f2 P fd82e4a300fb4c6caea9afdeec0c4147: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:08:43.230283 27152 ts_tablet_manager.cc:1918] T 1df0669b69ac461d9f77a40be73312f2 P 25f62de56e854bc49c156b4785824a5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:08:43.230476 27152 log.cc:1196] T 1df0669b69ac461d9f77a40be73312f2 P 25f62de56e854bc49c156b4785824a5d: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-2/wal/wals/1df0669b69ac461d9f77a40be73312f2
I20260812 07:08:43.231191 27152 ts_tablet_manager.cc:1939] T 1df0669b69ac461d9f77a40be73312f2 P 25f62de56e854bc49c156b4785824a5d: Deleting consensus metadata
I20260812 07:08:43.232508 27151 ts_tablet_manager.cc:1918] T 1df0669b69ac461d9f77a40be73312f2 P b7f98ec9cb3540c5aa3add4778180e02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:08:43.232782 27151 log.cc:1196] T 1df0669b69ac461d9f77a40be73312f2 P b7f98ec9cb3540c5aa3add4778180e02: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-1/wal/wals/1df0669b69ac461d9f77a40be73312f2
I20260812 07:08:43.233306 26565 catalog_manager.cc:4841] TS 25f62de56e854bc49c156b4785824a5d (127.6.61.3:42583): tablet 1df0669b69ac461d9f77a40be73312f2 (table test-table [id=8d037b3a1cab442ca8521e34b645f238]) successfully deleted
I20260812 07:08:43.233460 27151 ts_tablet_manager.cc:1939] T 1df0669b69ac461d9f77a40be73312f2 P b7f98ec9cb3540c5aa3add4778180e02: Deleting consensus metadata
I20260812 07:08:43.235327 26566 catalog_manager.cc:4841] TS b7f98ec9cb3540c5aa3add4778180e02 (127.6.61.2:37589): tablet 1df0669b69ac461d9f77a40be73312f2 (table test-table [id=8d037b3a1cab442ca8521e34b645f238]) successfully deleted
I20260812 07:08:43.239111 27153 ts_tablet_manager.cc:1918] T 1df0669b69ac461d9f77a40be73312f2 P fd82e4a300fb4c6caea9afdeec0c4147: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:08:43.239316 27153 log.cc:1196] T 1df0669b69ac461d9f77a40be73312f2 P fd82e4a300fb4c6caea9afdeec0c4147: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518208686283-6388-0/minicluster-data/ts-0/wal/wals/1df0669b69ac461d9f77a40be73312f2
Aug 12 07:08:43 dist-test-slave-69rg krb5kdc[26528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.1@KRBTEST.COM
I20260812 07:08:43.240000 27153 ts_tablet_manager.cc:1939] T 1df0669b69ac461d9f77a40be73312f2 P fd82e4a300fb4c6caea9afdeec0c4147: Deleting consensus metadata
I20260812 07:08:43.242398 26565 catalog_manager.cc:4841] TS fd82e4a300fb4c6caea9afdeec0c4147 (127.6.61.1:34001): tablet 1df0669b69ac461d9f77a40be73312f2 (table test-table [id=8d037b3a1cab442ca8521e34b645f238]) successfully deleted
I20260812 07:08:43.247704 27119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:43.219136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53872 (local address 127.6.61.1:34001)
0812 07:08:43.219370 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:43.219377 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:43.223666 (+ 4289us) server_negotiation.cc:407] Connection header received
0812 07:08:43.228526 (+ 4860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:43.228533 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:43.228631 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:43.228710 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:43.230812 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:43.231977 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:43.234107 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:43.234838 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:43.234988 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:43.241106 (+ 6118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:43.241122 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:43.241162 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:08:43.241216 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:43.244499 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:43.245409 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:43.245416 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:43.245419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:43.245552 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:43.246003 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:43.246007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:43.246009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:43.246420 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:43.246652 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:43.247065 (+ 413us) server_negotiation.cc:299] Negotiation successful
0812 07:08:43.247251 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 07:08:43.248472 26794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:08:43.350104 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 26656
I20260812 07:08:43.381995 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 26806
Aug 12 07:08:43 dist-test-slave-69rg krb5kdc[26528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.2@KRBTEST.COM
I20260812 07:08:43.402468 27114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:43.382923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34884 (local address 127.6.61.2:37589)
0812 07:08:43.383121 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:43.383126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:43.383216 (+ 90us) server_negotiation.cc:407] Connection header received
0812 07:08:43.383491 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:43.383496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:43.383576 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:43.383695 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:43.385765 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:43.386863 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:43.388954 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:43.389476 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:43.389593 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:43.395587 (+ 5994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:43.395606 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:43.395625 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:08:43.395694 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:43.399177 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:43.400153 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:43.400162 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:43.400164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:43.400329 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:43.400762 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:43.400765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:43.400767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:43.401191 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:43.401494 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:43.401797 (+ 303us) server_negotiation.cc:299] Negotiation successful
0812 07:08:43.402052 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 07:08:43.403286 26942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:08:43.517524 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 26806
I20260812 07:08:43.538704 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 26958
Aug 12 07:08:43 dist-test-slave-69rg krb5kdc[26528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.3@KRBTEST.COM
I20260812 07:08:43.555307 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:43.539527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42922 (local address 127.6.61.3:42583)
0812 07:08:43.539714 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:43.539719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:43.539738 (+ 19us) server_negotiation.cc:407] Connection header received
0812 07:08:43.539864 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:43.539868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:43.539948 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:43.540040 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:43.541573 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:43.542472 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:43.544076 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:43.544519 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:43.544621 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:43.549322 (+ 4701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:43.549337 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:43.549355 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:08:43.549390 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:43.552232 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:43.552988 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:43.552996 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:43.552999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:43.553165 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:43.553591 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:43.553594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:43.553596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:43.554252 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:43.554503 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:43.554825 (+ 322us) server_negotiation.cc:299] Negotiation successful
0812 07:08:43.555019 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 07:08:43.556058 27093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:08:43.647078 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 26958
I20260812 07:08:43.669991 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 26544
I20260812 07:08:43.671325 26616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:08:43.784514 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 26544
2026-08-12T07:08:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4850 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-12T07:08:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:08:43Z Disabled control of system clock
I20260812 07:08:43.887957 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:46583
--webserver_interface=127.6.61.62
--webserver_port=0
--builtin_ntp_servers=127.6.61.20:36449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:46583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 07:08:44.255964 27163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:08:44.256278 27163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:08:44.256433 27163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:08:44.263911 27163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:08:44.264019 27163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:08:44.264104 27163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:08:44.264194 27163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:08:44.264261 27163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:08:44.273523 27163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:36449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:46583
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:46583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:08:44.274930 27163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:08:44.276741 27163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:08:44.288847 27168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:44.288918 27169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:44.289371 27171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:08:44.290269 27163 server_base.cc:1034] running on GCE node
I20260812 07:08:44.291139 27163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:08:44.292783 27163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:08:44.294046 27163 hybrid_clock.cc:648] HybridClock initialized: now 1786518524293982 us; error 48 us; skew 500 ppm
I20260812 07:08:44.294445 27163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 07:08:44.296762 27163 webserver.cc:460] Webserver started at http://127.6.61.62:33381/ using document root <none> and password file <none>
I20260812 07:08:44.297520 27163 fs_manager.cc:359] Metadata directory not provided
I20260812 07:08:44.297685 27163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:08:44.298090 27163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:08:44.301138 27163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "c666142015d0467080a07128d22ef6c2"
format_stamp: "Formatted at 2026-08-12 07:08:44 on dist-test-slave-69rg"
server_key: "ea4c4a360abfda9dc040241494f635ff"
server_key_iv: "66c2e779f3b2d1a85289fd50deadef36"
server_key_version: "encryptionkey@0"
I20260812 07:08:44.302062 27163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "c666142015d0467080a07128d22ef6c2"
format_stamp: "Formatted at 2026-08-12 07:08:44 on dist-test-slave-69rg"
server_key: "ea4c4a360abfda9dc040241494f635ff"
server_key_iv: "66c2e779f3b2d1a85289fd50deadef36"
server_key_version: "encryptionkey@0"
I20260812 07:08:44.307471 27163 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 07:08:44.311425 27177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:44.312924 27163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:08:44.313167 27163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "c666142015d0467080a07128d22ef6c2"
format_stamp: "Formatted at 2026-08-12 07:08:44 on dist-test-slave-69rg"
server_key: "ea4c4a360abfda9dc040241494f635ff"
server_key_iv: "66c2e779f3b2d1a85289fd50deadef36"
server_key_version: "encryptionkey@0"
I20260812 07:08:44.313458 27163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:08:44.342283 27163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:08:44.343289 27163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:08:44.343641 27163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:08:44.364746 27163 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:46583
I20260812 07:08:44.364765 27237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:46583 every 8 connection(s)
I20260812 07:08:44.366218 27163 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:08:44.370236 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:08:44.372030 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 27163
I20260812 07:08:44.372256 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:08:44.372732 6388 external_mini_cluster.cc:1428] Setting key c066601c2095f0b7ea6a0e3ebedc1fd5
I20260812 07:08:44.385390 27238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c666142015d0467080a07128d22ef6c2: Bootstrap starting.
I20260812 07:08:44.389743 27238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c666142015d0467080a07128d22ef6c2: Neither blocks nor log segments found. Creating new log.
I20260812 07:08:44.390910 27242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:44.374891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33678 (local address 127.6.61.62:46583)
0812 07:08:44.376125 (+ 1234us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:44.376141 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:44.376194 (+ 53us) server_negotiation.cc:407] Connection header received
0812 07:08:44.378739 (+ 2545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:44.378774 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:44.379430 (+ 656us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:44.379917 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:44.380535 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:44.382732 (+ 2197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:44.384460 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:44.385932 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:44.386162 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:44.387678 (+ 1516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:44.387692 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:44.387713 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:08:44.388498 (+ 785us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:44.388643 (+ 145us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:08:44.388681 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:44.388806 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:44.389089 (+ 283us) server_negotiation.cc:299] Negotiation successful
0812 07:08:44.389463 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":417,"threads_started":1}
I20260812 07:08:44.391299 27238 log.cc:824] T 00000000000000000000000000000000 P c666142015d0467080a07128d22ef6c2: Log is configured to *not* fsync() on all Append() calls
I20260812 07:08:44.395087 27238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c666142015d0467080a07128d22ef6c2: No bootstrap required, opened a new log
I20260812 07:08:44.403474 27238 raft_consensus.cc:348] T 00000000000000000000000000000000 P c666142015d0467080a07128d22ef6c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c666142015d0467080a07128d22ef6c2" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 46583 } }
I20260812 07:08:44.403961 27238 raft_consensus.cc:374] T 00000000000000000000000000000000 P c666142015d0467080a07128d22ef6c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:08:44.404080 27238 raft_consensus.cc:729] T 00000000000000000000000000000000 P c666142015d0467080a07128d22ef6c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c666142015d0467080a07128d22ef6c2, State: Initialized, Role: FOLLOWER
I20260812 07:08:44.404704 27238 consensus_queue.cc:260] T 00000000000000000000000000000000 P c666142015d0467080a07128d22ef6c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c666142015d0467080a07128d22ef6c2" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 46583 } }
I20260812 07:08:44.404978 27238 raft_consensus.cc:388] T 00000000000000000000000000000000 P c666142015d0467080a07128d22ef6c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:08:44.405144 27238 raft_consensus.cc:482] T 00000000000000000000000000000000 P c666142015d0467080a07128d22ef6c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:08:44.405339 27238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c666142015d0467080a07128d22ef6c2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:08:44.407692 27238 raft_consensus.cc:504] T 00000000000000000000000000000000 P c666142015d0467080a07128d22ef6c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c666142015d0467080a07128d22ef6c2" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 46583 } }
I20260812 07:08:44.408241 27238 leader_election.cc:302] T 00000000000000000000000000000000 P c666142015d0467080a07128d22ef6c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c666142015d0467080a07128d22ef6c2; no voters:
I20260812 07:08:44.409015 27238 leader_election.cc:288] T 00000000000000000000000000000000 P c666142015d0467080a07128d22ef6c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:08:44.409144 27244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c666142015d0467080a07128d22ef6c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:08:44.409989 27244 raft_consensus.cc:686] T 00000000000000000000000000000000 P c666142015d0467080a07128d22ef6c2 [term 1 LEADER]: Becoming Leader. State: Replica: c666142015d0467080a07128d22ef6c2, State: Running, Role: LEADER
I20260812 07:08:44.410569 27244 consensus_queue.cc:237] T 00000000000000000000000000000000 P c666142015d0467080a07128d22ef6c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c666142015d0467080a07128d22ef6c2" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 46583 } }
I20260812 07:08:44.411608 27238 sys_catalog.cc:564] T 00000000000000000000000000000000 P c666142015d0467080a07128d22ef6c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:08:44.413991 27246 sys_catalog.cc:455] T 00000000000000000000000000000000 P c666142015d0467080a07128d22ef6c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c666142015d0467080a07128d22ef6c2. Latest consensus state: current_term: 1 leader_uuid: "c666142015d0467080a07128d22ef6c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c666142015d0467080a07128d22ef6c2" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 46583 } } }
I20260812 07:08:44.414294 27245 sys_catalog.cc:455] T 00000000000000000000000000000000 P c666142015d0467080a07128d22ef6c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c666142015d0467080a07128d22ef6c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c666142015d0467080a07128d22ef6c2" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 46583 } } }
I20260812 07:08:44.414505 27246 sys_catalog.cc:458] T 00000000000000000000000000000000 P c666142015d0467080a07128d22ef6c2 [sys.catalog]: This master's current role is: LEADER
I20260812 07:08:44.419735 27245 sys_catalog.cc:458] T 00000000000000000000000000000000 P c666142015d0467080a07128d22ef6c2 [sys.catalog]: This master's current role is: LEADER
I20260812 07:08:44.423388 27252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:08:44.431509 27252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:08:44.442287 27252 catalog_manager.cc:1347] Generated new cluster ID: ab45fd3c99454908abc2ec71e303bd41
I20260812 07:08:44.442452 27252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:08:44.462564 27252 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:08:44.463995 27252 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:08:44.470665 27252 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c666142015d0467080a07128d22ef6c2: Generated new TSK 0
I20260812 07:08:44.471503 27252 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:08:44.476903 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.1:0
--local_ip_for_outbound_sockets=127.6.61.1
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:46583
--builtin_ntp_servers=127.6.61.20:36449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 07:08:44.836485 27264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:08:44.836918 27264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:08:44.837096 27264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:08:44.844655 27264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:08:44.844853 27264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:08:44.845067 27264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.1
I20260812 07:08:44.855391 27264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:36449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:46583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.1
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:08:44.856844 27264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:08:44.858860 27264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:08:44.872947 27269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:44.872916 27272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:44.872888 27270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:08:44.873668 27264 server_base.cc:1034] running on GCE node
I20260812 07:08:44.874380 27264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:08:44.875377 27264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:08:44.876619 27264 hybrid_clock.cc:648] HybridClock initialized: now 1786518524876554 us; error 43 us; skew 500 ppm
I20260812 07:08:44.876986 27264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 07:08:44.879503 27264 webserver.cc:460] Webserver started at http://127.6.61.1:39851/ using document root <none> and password file <none>
I20260812 07:08:44.880240 27264 fs_manager.cc:359] Metadata directory not provided
I20260812 07:08:44.880362 27264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:08:44.880730 27264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:08:44.883880 27264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-0/data/instance:
uuid: "105f224cc0d6477d9220d8a5d8c60bb1"
format_stamp: "Formatted at 2026-08-12 07:08:44 on dist-test-slave-69rg"
server_key: "91ae40f246e057b580d0b8b082f1ab50"
server_key_iv: "7aa96c5caaaf66d087311c231dafd9b5"
server_key_version: "encryptionkey@0"
I20260812 07:08:44.884703 27264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance:
uuid: "105f224cc0d6477d9220d8a5d8c60bb1"
format_stamp: "Formatted at 2026-08-12 07:08:44 on dist-test-slave-69rg"
server_key: "91ae40f246e057b580d0b8b082f1ab50"
server_key_iv: "7aa96c5caaaf66d087311c231dafd9b5"
server_key_version: "encryptionkey@0"
I20260812 07:08:44.889946 27264 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 07:08:44.893716 27278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:44.895190 27264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:08:44.895397 27264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-0/wal
uuid: "105f224cc0d6477d9220d8a5d8c60bb1"
format_stamp: "Formatted at 2026-08-12 07:08:44 on dist-test-slave-69rg"
server_key: "91ae40f246e057b580d0b8b082f1ab50"
server_key_iv: "7aa96c5caaaf66d087311c231dafd9b5"
server_key_version: "encryptionkey@0"
I20260812 07:08:44.895644 27264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:08:44.916306 27264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:08:44.917229 27264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:08:44.917551 27264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:08:44.918776 27264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:08:44.920398 27264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:08:44.920522 27264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:44.920667 27264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:08:44.920761 27264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:44.957180 27264 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.1:33603
I20260812 07:08:44.957248 27404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.1:33603 every 8 connection(s)
I20260812 07:08:44.958691 27264 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
I20260812 07:08:44.965332 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 27264
I20260812 07:08:44.965571 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance
I20260812 07:08:44.966267 6388 external_mini_cluster.cc:1428] Setting key bb846ad86cca7d9faafa929aa8db817a
I20260812 07:08:44.969889 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.2:0
--local_ip_for_outbound_sockets=127.6.61.2
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:46583
--builtin_ntp_servers=127.6.61.20:36449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 07:08:44.974731 27407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:44.961547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:54243 (local address 127.6.61.62:46583)
0812 07:08:44.962087 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:44.962093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:44.963468 (+ 1375us) server_negotiation.cc:407] Connection header received
0812 07:08:44.965575 (+ 2107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:44.965584 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:44.965733 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:44.965912 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:44.967563 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:44.968703 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:44.971420 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:44.971850 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:44.971963 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:44.973775 (+ 1812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:44.973781 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:44.973784 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:08:44.973806 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:44.973919 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:08:44.973935 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:44.974028 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:44.974340 (+ 312us) server_negotiation.cc:299] Negotiation successful
0812 07:08:44.974439 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":285,"threads_started":1}
I20260812 07:08:44.976266 27408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:44.962286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:46583 (local address 127.6.61.1:54243)
0812 07:08:44.963284 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:44.963334 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:44.965181 (+ 1847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:44.966168 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:44.966180 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:44.966315 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:08:44.967322 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:44.967353 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:44.968917 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:44.968923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:44.971250 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:44.971272 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:44.972130 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:44.972141 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:44.972490 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:44.973348 (+ 858us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:08:44.973354 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:08:44.973414 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:08:44.973541 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:08:44.974052 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:08:44.974067 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:08:44.974078 (+ 11us) client_negotiation.cc:769] Sending connection context
0812 07:08:44.974288 (+ 210us) client_negotiation.cc:240] Negotiation successful
0812 07:08:44.974766 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":309,"threads_started":1}
I20260812 07:08:44.978243 27405 heartbeater.cc:347] Connected to a master server at 127.6.61.62:46583
I20260812 07:08:44.978610 27405 heartbeater.cc:464] Registering TS with master...
I20260812 07:08:44.979646 27405 heartbeater.cc:511] Master 127.6.61.62:46583 requested a full tablet report, sending...
I20260812 07:08:44.982489 27195 ts_manager.cc:194] Registered new tserver with Master: 105f224cc0d6477d9220d8a5d8c60bb1 (127.6.61.1:33603)
I20260812 07:08:44.984539 27195 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.61.1:54243
W20260812 07:08:45.320685 27409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:08:45.320988 27409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:08:45.321126 27409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:08:45.328576 27409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:08:45.328688 27409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:08:45.328868 27409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.2
I20260812 07:08:45.338272 27409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:36449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:46583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.2
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:08:45.339629 27409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:08:45.341431 27409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:08:45.355721 27414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:45.355746 27415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:45.355947 27417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:08:45.356691 27409 server_base.cc:1034] running on GCE node
I20260812 07:08:45.357391 27409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:08:45.358497 27409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:08:45.359762 27409 hybrid_clock.cc:648] HybridClock initialized: now 1786518525359701 us; error 51 us; skew 500 ppm
I20260812 07:08:45.360132 27409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 07:08:45.362612 27409 webserver.cc:460] Webserver started at http://127.6.61.2:35531/ using document root <none> and password file <none>
I20260812 07:08:45.363350 27409 fs_manager.cc:359] Metadata directory not provided
I20260812 07:08:45.363489 27409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:08:45.363829 27409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:08:45.366892 27409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-1/data/instance:
uuid: "eef84b63f07c492b89734c9fe9a6bf73"
format_stamp: "Formatted at 2026-08-12 07:08:45 on dist-test-slave-69rg"
server_key: "bb791a703cd079f78920df966f68b6c3"
server_key_iv: "8ef5e8b8c70563d0079643c50b86ee3c"
server_key_version: "encryptionkey@0"
I20260812 07:08:45.367735 27409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance:
uuid: "eef84b63f07c492b89734c9fe9a6bf73"
format_stamp: "Formatted at 2026-08-12 07:08:45 on dist-test-slave-69rg"
server_key: "bb791a703cd079f78920df966f68b6c3"
server_key_iv: "8ef5e8b8c70563d0079643c50b86ee3c"
server_key_version: "encryptionkey@0"
I20260812 07:08:45.373266 27409 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 07:08:45.377298 27423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:45.378844 27409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:08:45.379038 27409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-1/wal
uuid: "eef84b63f07c492b89734c9fe9a6bf73"
format_stamp: "Formatted at 2026-08-12 07:08:45 on dist-test-slave-69rg"
server_key: "bb791a703cd079f78920df966f68b6c3"
server_key_iv: "8ef5e8b8c70563d0079643c50b86ee3c"
server_key_version: "encryptionkey@0"
I20260812 07:08:45.379284 27409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:08:45.393735 27409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:08:45.394791 27409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:08:45.395049 27409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:08:45.396063 27409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:08:45.397639 27409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:08:45.397739 27409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:45.397893 27409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:08:45.397960 27409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260812 07:08:45.438041 27409 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.2:43655
I20260812 07:08:45.438122 27549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.2:43655 every 8 connection(s)
I20260812 07:08:45.439689 27409 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
I20260812 07:08:45.446341 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 27409
I20260812 07:08:45.446544 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance
I20260812 07:08:45.447114 6388 external_mini_cluster.cc:1428] Setting key 9153305a16fa53dda30af5bc45429ce9
I20260812 07:08:45.450908 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.3:0
--local_ip_for_outbound_sockets=127.6.61.3
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:46583
--builtin_ntp_servers=127.6.61.20:36449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 07:08:45.457690 27407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:45.443099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:37223 (local address 127.6.61.62:46583)
0812 07:08:45.443460 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:45.443465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:45.444799 (+ 1334us) server_negotiation.cc:407] Connection header received
0812 07:08:45.447698 (+ 2899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:45.447707 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:45.447836 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:45.447988 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:45.449660 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:45.450813 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:45.453747 (+ 2934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:45.454413 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:45.454540 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:45.456562 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:45.456568 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:45.456571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:08:45.456595 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:45.456635 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:08:45.456649 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:45.456737 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:45.457127 (+ 390us) server_negotiation.cc:299] Negotiation successful
0812 07:08:45.457270 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 07:08:45.460816 27552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:45.443555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:46583 (local address 127.6.61.2:37223)
0812 07:08:45.444614 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:45.444662 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:45.447198 (+ 2536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:45.448312 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:45.448327 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:45.448492 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:08:45.449369 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:45.449403 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:45.451022 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:45.451028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:45.453491 (+ 2463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:45.453518 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:45.454703 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:45.454714 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:45.455116 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:45.456047 (+ 931us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:08:45.456053 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:08:45.456121 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:08:45.456253 (+ 132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:08:45.456775 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:08:45.456790 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:08:45.456802 (+ 12us) client_negotiation.cc:769] Sending connection context
0812 07:08:45.457041 (+ 239us) client_negotiation.cc:240] Negotiation successful
0812 07:08:45.458281 (+ 1240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":383,"threads_started":1}
I20260812 07:08:45.464668 27550 heartbeater.cc:347] Connected to a master server at 127.6.61.62:46583
I20260812 07:08:45.465179 27550 heartbeater.cc:464] Registering TS with master...
I20260812 07:08:45.466364 27550 heartbeater.cc:511] Master 127.6.61.62:46583 requested a full tablet report, sending...
I20260812 07:08:45.468618 27195 ts_manager.cc:194] Registered new tserver with Master: eef84b63f07c492b89734c9fe9a6bf73 (127.6.61.2:43655)
I20260812 07:08:45.469787 27195 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.61.2:37223
W20260812 07:08:45.831786 27553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:08:45.832127 27553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:08:45.832267 27553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:08:45.840137 27553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:08:45.840281 27553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:08:45.840469 27553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.3
I20260812 07:08:45.850695 27553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:36449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:46583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.3
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:08:45.852239 27553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:08:45.854321 27553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:08:45.870667 27559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:45.870683 27560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:45.870714 27562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:08:45.872112 27553 server_base.cc:1034] running on GCE node
I20260812 07:08:45.872881 27553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:08:45.874121 27553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:08:45.875365 27553 hybrid_clock.cc:648] HybridClock initialized: now 1786518525875287 us; error 63 us; skew 500 ppm
I20260812 07:08:45.875761 27553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 07:08:45.878580 27553 webserver.cc:460] Webserver started at http://127.6.61.3:41661/ using document root <none> and password file <none>
I20260812 07:08:45.879350 27553 fs_manager.cc:359] Metadata directory not provided
I20260812 07:08:45.879446 27553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:08:45.879829 27553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:08:45.883009 27553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-2/data/instance:
uuid: "1072d373e945448d96fb315bca7ed76a"
format_stamp: "Formatted at 2026-08-12 07:08:45 on dist-test-slave-69rg"
server_key: "5fd26b2e078859e562b657a575178cb5"
server_key_iv: "4b7aa39db07ca71e1448ec2caf0ab4ef"
server_key_version: "encryptionkey@0"
I20260812 07:08:45.884014 27553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance:
uuid: "1072d373e945448d96fb315bca7ed76a"
format_stamp: "Formatted at 2026-08-12 07:08:45 on dist-test-slave-69rg"
server_key: "5fd26b2e078859e562b657a575178cb5"
server_key_iv: "4b7aa39db07ca71e1448ec2caf0ab4ef"
server_key_version: "encryptionkey@0"
I20260812 07:08:45.890105 27553 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 07:08:45.894580 27568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:45.896209 27553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 07:08:45.896404 27553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-2/wal
uuid: "1072d373e945448d96fb315bca7ed76a"
format_stamp: "Formatted at 2026-08-12 07:08:45 on dist-test-slave-69rg"
server_key: "5fd26b2e078859e562b657a575178cb5"
server_key_iv: "4b7aa39db07ca71e1448ec2caf0ab4ef"
server_key_version: "encryptionkey@0"
I20260812 07:08:45.896656 27553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:08:45.919679 27553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:08:45.920660 27553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:08:45.920987 27553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:08:45.922199 27553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:08:45.923802 27553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:08:45.923892 27553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:45.924047 27553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:08:45.924126 27553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:45.962565 27553 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.3:33833
I20260812 07:08:45.962774 27695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.3:33833 every 8 connection(s)
I20260812 07:08:45.964411 27553 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
I20260812 07:08:45.965317 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 27553
I20260812 07:08:45.965515 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance
I20260812 07:08:45.966187 6388 external_mini_cluster.cc:1428] Setting key 75f841042da273cf489c7d8f5f3da69f
I20260812 07:08:45.979243 27698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:45.967592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:59103 (local address 127.6.61.62:46583)
0812 07:08:45.968143 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:45.968149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:45.969621 (+ 1472us) server_negotiation.cc:407] Connection header received
0812 07:08:45.971672 (+ 2051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:45.971677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:45.971760 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:45.971863 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:45.973055 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:45.973997 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:45.975953 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:45.976408 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:45.976508 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:45.977976 (+ 1468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:45.977981 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:45.977984 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:08:45.978004 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:45.978044 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:08:45.978056 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:45.978143 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:45.978617 (+ 474us) server_negotiation.cc:299] Negotiation successful
0812 07:08:45.978805 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":284,"threads_started":1}
I20260812 07:08:45.979879 27699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:45.968302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:46583 (local address 127.6.61.3:59103)
0812 07:08:45.969442 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:45.969497 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:45.971390 (+ 1893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:45.972064 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:45.972073 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:45.972176 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:08:45.972815 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:45.972834 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:45.974259 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:45.974264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:45.975795 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:45.975810 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:45.976619 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:45.976624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:45.976852 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:45.977576 (+ 724us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:08:45.977579 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:08:45.977622 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:08:45.977712 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:08:45.978161 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:08:45.978169 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:08:45.978176 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 07:08:45.978395 (+ 219us) client_negotiation.cc:240] Negotiation successful
0812 07:08:45.978732 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":409,"threads_started":1}
I20260812 07:08:45.981654 27696 heartbeater.cc:347] Connected to a master server at 127.6.61.62:46583
I20260812 07:08:45.982098 27696 heartbeater.cc:464] Registering TS with master...
I20260812 07:08:45.983146 27696 heartbeater.cc:511] Master 127.6.61.62:46583 requested a full tablet report, sending...
I20260812 07:08:45.984895 27195 ts_manager.cc:194] Registered new tserver with Master: 1072d373e945448d96fb315bca7ed76a (127.6.61.3:33833)
I20260812 07:08:45.985864 27195 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.61.3:59103
I20260812 07:08:45.987829 27405 heartbeater.cc:503] Master 127.6.61.62:46583 was elected leader, sending a full tablet report...
I20260812 07:08:45.995743 6388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 07:08:46.007494 27698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:46.000504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33682 (local address 127.6.61.62:46583)
0812 07:08:46.000786 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:46.000793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:46.001178 (+ 385us) server_negotiation.cc:407] Connection header received
0812 07:08:46.001339 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:46.001344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:46.001442 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:46.001543 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:46.002114 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:46.002931 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:46.004629 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:46.005125 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:46.005256 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:46.006524 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:46.006530 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:46.006533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:08:46.006554 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:46.006587 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:08:46.006600 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:46.006684 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:46.007104 (+ 420us) server_negotiation.cc:299] Negotiation successful
0812 07:08:46.007205 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260812 07:08:46.012264 27195 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:08:46.020026 27195 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:08:46.050125 27709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:46.034201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38546 (local address 127.6.61.2:43655)
0812 07:08:46.035302 (+ 1101us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:46.035319 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:46.036556 (+ 1237us) server_negotiation.cc:407] Connection header received
0812 07:08:46.036964 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:46.036971 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:46.037234 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:46.037472 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:08:46.038815 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:46.041061 (+ 2246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:46.045094 (+ 4033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:46.047356 (+ 2262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:46.047512 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:46.047883 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:46.049537 (+ 1654us) server_negotiation.cc:299] Negotiation successful
0812 07:08:46.049764 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":558,"threads_started":1}
I20260812 07:08:46.051100 27712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:46.034278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:43655 (local address 127.0.0.1:38546)
0812 07:08:46.036389 (+ 2111us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:46.036421 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:46.036618 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:46.037863 (+ 1245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:46.037875 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:46.037931 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:08:46.038536 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:46.038556 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:46.041326 (+ 2770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:46.041335 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:46.044758 (+ 3423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:46.044787 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:46.048130 (+ 3343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:46.048139 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:46.049172 (+ 1033us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:46.049285 (+ 113us) client_negotiation.cc:769] Sending connection context
0812 07:08:46.050439 (+ 1154us) client_negotiation.cc:240] Negotiation successful
0812 07:08:46.050672 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1866,"mutex_wait_us":610,"thread_start_us":1691,"threads_started":1}
I20260812 07:08:46.052037 27710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:46.034436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56078 (local address 127.6.61.3:33833)
0812 07:08:46.035922 (+ 1486us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:46.035943 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:46.036832 (+ 889us) server_negotiation.cc:407] Connection header received
0812 07:08:46.036942 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:46.036950 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:46.037254 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:46.037478 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:08:46.040377 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:46.044246 (+ 3869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:46.049516 (+ 5270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:46.050920 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:46.051084 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:46.051409 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:46.051527 (+ 118us) server_negotiation.cc:299] Negotiation successful
0812 07:08:46.051703 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1296,"thread_start_us":1113,"threads_started":1}
I20260812 07:08:46.051977 27711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:46.034677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:33833 (local address 127.0.0.1:56078)
0812 07:08:46.036392 (+ 1715us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:46.036424 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:46.036623 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:46.039459 (+ 2836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:46.039467 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:46.039505 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:08:46.040083 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:46.040100 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:46.044470 (+ 4370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:46.044479 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:46.049256 (+ 4777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:46.049275 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:46.051148 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:46.051155 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:46.051342 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:46.051388 (+ 46us) client_negotiation.cc:769] Sending connection context
0812 07:08:46.051489 (+ 101us) client_negotiation.cc:240] Negotiation successful
0812 07:08:46.051641 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1551,"mutex_wait_us":34,"thread_start_us":1916,"threads_started":1}
I20260812 07:08:46.055709 27480 tablet_service.cc:1467] Processing CreateTablet for tablet 20920e918fc54b4c9856d71bfb474613 (DEFAULT_TABLE table=test-table [id=86c87d191b3b4e78887976a604430eb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:08:46.057035 27480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20920e918fc54b4c9856d71bfb474613. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:08:46.056718 27624 tablet_service.cc:1467] Processing CreateTablet for tablet 20920e918fc54b4c9856d71bfb474613 (DEFAULT_TABLE table=test-table [id=86c87d191b3b4e78887976a604430eb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:08:46.058878 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:46.036922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:33603 (local address 127.0.0.1:36694)
0812 07:08:46.049444 (+ 12522us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:46.049466 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:46.049602 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:46.050538 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:46.050543 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:46.050570 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:08:46.050970 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:46.050982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:46.053161 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:46.053170 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:46.055244 (+ 2074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:46.055262 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:46.057954 (+ 2692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:46.057962 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:46.058177 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:46.058219 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 07:08:46.058346 (+ 127us) client_negotiation.cc:240] Negotiation successful
0812 07:08:46.058494 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12352,"thread_start_us":12172,"threads_started":1}
I20260812 07:08:46.059104 27715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:46.037384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36694 (local address 127.6.61.1:33603)
0812 07:08:46.049013 (+ 11629us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:46.049026 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:46.049568 (+ 542us) server_negotiation.cc:407] Connection header received
0812 07:08:46.049792 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:46.049803 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:46.050136 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:46.050368 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:08:46.051347 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:46.052967 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:46.055672 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:46.057670 (+ 1998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:46.057906 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:46.058252 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:46.058428 (+ 176us) server_negotiation.cc:299] Negotiation successful
0812 07:08:46.058637 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11430,"thread_start_us":11268,"threads_started":1}
I20260812 07:08:46.059931 27624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20920e918fc54b4c9856d71bfb474613. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:08:46.067373 27332 tablet_service.cc:1467] Processing CreateTablet for tablet 20920e918fc54b4c9856d71bfb474613 (DEFAULT_TABLE table=test-table [id=86c87d191b3b4e78887976a604430eb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:08:46.069334 27332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20920e918fc54b4c9856d71bfb474613. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:08:46.075290 27718 tablet_bootstrap.cc:492] T 20920e918fc54b4c9856d71bfb474613 P eef84b63f07c492b89734c9fe9a6bf73: Bootstrap starting.
I20260812 07:08:46.077546 27719 tablet_bootstrap.cc:492] T 20920e918fc54b4c9856d71bfb474613 P 1072d373e945448d96fb315bca7ed76a: Bootstrap starting.
I20260812 07:08:46.080475 27718 tablet_bootstrap.cc:654] T 20920e918fc54b4c9856d71bfb474613 P eef84b63f07c492b89734c9fe9a6bf73: Neither blocks nor log segments found. Creating new log.
I20260812 07:08:46.081357 27719 tablet_bootstrap.cc:654] T 20920e918fc54b4c9856d71bfb474613 P 1072d373e945448d96fb315bca7ed76a: Neither blocks nor log segments found. Creating new log.
I20260812 07:08:46.082098 27718 log.cc:824] T 20920e918fc54b4c9856d71bfb474613 P eef84b63f07c492b89734c9fe9a6bf73: Log is configured to *not* fsync() on all Append() calls
I20260812 07:08:46.082551 27719 log.cc:824] T 20920e918fc54b4c9856d71bfb474613 P 1072d373e945448d96fb315bca7ed76a: Log is configured to *not* fsync() on all Append() calls
I20260812 07:08:46.085237 27723 tablet_bootstrap.cc:492] T 20920e918fc54b4c9856d71bfb474613 P 105f224cc0d6477d9220d8a5d8c60bb1: Bootstrap starting.
I20260812 07:08:46.085932 27718 tablet_bootstrap.cc:492] T 20920e918fc54b4c9856d71bfb474613 P eef84b63f07c492b89734c9fe9a6bf73: No bootstrap required, opened a new log
I20260812 07:08:46.086256 27718 ts_tablet_manager.cc:1397] T 20920e918fc54b4c9856d71bfb474613 P eef84b63f07c492b89734c9fe9a6bf73: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260812 07:08:46.086311 27719 tablet_bootstrap.cc:492] T 20920e918fc54b4c9856d71bfb474613 P 1072d373e945448d96fb315bca7ed76a: No bootstrap required, opened a new log
I20260812 07:08:46.086592 27719 ts_tablet_manager.cc:1397] T 20920e918fc54b4c9856d71bfb474613 P 1072d373e945448d96fb315bca7ed76a: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260812 07:08:46.089440 27723 tablet_bootstrap.cc:654] T 20920e918fc54b4c9856d71bfb474613 P 105f224cc0d6477d9220d8a5d8c60bb1: Neither blocks nor log segments found. Creating new log.
I20260812 07:08:46.090955 27723 log.cc:824] T 20920e918fc54b4c9856d71bfb474613 P 105f224cc0d6477d9220d8a5d8c60bb1: Log is configured to *not* fsync() on all Append() calls
I20260812 07:08:46.095057 27719 raft_consensus.cc:348] T 20920e918fc54b4c9856d71bfb474613 P 1072d373e945448d96fb315bca7ed76a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef84b63f07c492b89734c9fe9a6bf73" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 43655 } } peers { permanent_uuid: "1072d373e945448d96fb315bca7ed76a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33833 } } peers { permanent_uuid: "105f224cc0d6477d9220d8a5d8c60bb1" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 33603 } }
I20260812 07:08:46.095402 27723 tablet_bootstrap.cc:492] T 20920e918fc54b4c9856d71bfb474613 P 105f224cc0d6477d9220d8a5d8c60bb1: No bootstrap required, opened a new log
I20260812 07:08:46.095510 27719 raft_consensus.cc:374] T 20920e918fc54b4c9856d71bfb474613 P 1072d373e945448d96fb315bca7ed76a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:08:46.095674 27719 raft_consensus.cc:729] T 20920e918fc54b4c9856d71bfb474613 P 1072d373e945448d96fb315bca7ed76a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1072d373e945448d96fb315bca7ed76a, State: Initialized, Role: FOLLOWER
I20260812 07:08:46.095711 27723 ts_tablet_manager.cc:1397] T 20920e918fc54b4c9856d71bfb474613 P 105f224cc0d6477d9220d8a5d8c60bb1: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260812 07:08:46.096517 27719 consensus_queue.cc:260] T 20920e918fc54b4c9856d71bfb474613 P 1072d373e945448d96fb315bca7ed76a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef84b63f07c492b89734c9fe9a6bf73" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 43655 } } peers { permanent_uuid: "1072d373e945448d96fb315bca7ed76a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33833 } } peers { permanent_uuid: "105f224cc0d6477d9220d8a5d8c60bb1" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 33603 } }
I20260812 07:08:46.097075 27718 raft_consensus.cc:348] T 20920e918fc54b4c9856d71bfb474613 P eef84b63f07c492b89734c9fe9a6bf73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef84b63f07c492b89734c9fe9a6bf73" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 43655 } } peers { permanent_uuid: "1072d373e945448d96fb315bca7ed76a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33833 } } peers { permanent_uuid: "105f224cc0d6477d9220d8a5d8c60bb1" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 33603 } }
I20260812 07:08:46.097877 27696 heartbeater.cc:503] Master 127.6.61.62:46583 was elected leader, sending a full tablet report...
I20260812 07:08:46.098560 27719 ts_tablet_manager.cc:1428] T 20920e918fc54b4c9856d71bfb474613 P 1072d373e945448d96fb315bca7ed76a: Time spent starting tablet: real 0.012s user 0.006s sys 0.006s
I20260812 07:08:46.098821 27718 raft_consensus.cc:374] T 20920e918fc54b4c9856d71bfb474613 P eef84b63f07c492b89734c9fe9a6bf73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:08:46.098999 27718 raft_consensus.cc:729] T 20920e918fc54b4c9856d71bfb474613 P eef84b63f07c492b89734c9fe9a6bf73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eef84b63f07c492b89734c9fe9a6bf73, State: Initialized, Role: FOLLOWER
I20260812 07:08:46.099814 27718 consensus_queue.cc:260] T 20920e918fc54b4c9856d71bfb474613 P eef84b63f07c492b89734c9fe9a6bf73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef84b63f07c492b89734c9fe9a6bf73" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 43655 } } peers { permanent_uuid: "1072d373e945448d96fb315bca7ed76a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33833 } } peers { permanent_uuid: "105f224cc0d6477d9220d8a5d8c60bb1" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 33603 } }
I20260812 07:08:46.101707 27718 ts_tablet_manager.cc:1428] T 20920e918fc54b4c9856d71bfb474613 P eef84b63f07c492b89734c9fe9a6bf73: Time spent starting tablet: real 0.015s user 0.008s sys 0.006s
I20260812 07:08:46.102213 27550 heartbeater.cc:503] Master 127.6.61.62:46583 was elected leader, sending a full tablet report...
I20260812 07:08:46.107928 27723 raft_consensus.cc:348] T 20920e918fc54b4c9856d71bfb474613 P 105f224cc0d6477d9220d8a5d8c60bb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef84b63f07c492b89734c9fe9a6bf73" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 43655 } } peers { permanent_uuid: "1072d373e945448d96fb315bca7ed76a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33833 } } peers { permanent_uuid: "105f224cc0d6477d9220d8a5d8c60bb1" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 33603 } }
I20260812 07:08:46.108402 27723 raft_consensus.cc:374] T 20920e918fc54b4c9856d71bfb474613 P 105f224cc0d6477d9220d8a5d8c60bb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:08:46.108482 27723 raft_consensus.cc:729] T 20920e918fc54b4c9856d71bfb474613 P 105f224cc0d6477d9220d8a5d8c60bb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 105f224cc0d6477d9220d8a5d8c60bb1, State: Initialized, Role: FOLLOWER
I20260812 07:08:46.109038 27723 consensus_queue.cc:260] T 20920e918fc54b4c9856d71bfb474613 P 105f224cc0d6477d9220d8a5d8c60bb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef84b63f07c492b89734c9fe9a6bf73" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 43655 } } peers { permanent_uuid: "1072d373e945448d96fb315bca7ed76a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33833 } } peers { permanent_uuid: "105f224cc0d6477d9220d8a5d8c60bb1" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 33603 } }
I20260812 07:08:46.110895 27723 ts_tablet_manager.cc:1428] T 20920e918fc54b4c9856d71bfb474613 P 105f224cc0d6477d9220d8a5d8c60bb1: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260812 07:08:46.115140 27725 raft_consensus.cc:482] T 20920e918fc54b4c9856d71bfb474613 P 1072d373e945448d96fb315bca7ed76a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:08:46.115515 27725 raft_consensus.cc:504] T 20920e918fc54b4c9856d71bfb474613 P 1072d373e945448d96fb315bca7ed76a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef84b63f07c492b89734c9fe9a6bf73" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 43655 } } peers { permanent_uuid: "1072d373e945448d96fb315bca7ed76a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33833 } } peers { permanent_uuid: "105f224cc0d6477d9220d8a5d8c60bb1" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 33603 } }
I20260812 07:08:46.118098 27725 leader_election.cc:288] T 20920e918fc54b4c9856d71bfb474613 P 1072d373e945448d96fb315bca7ed76a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eef84b63f07c492b89734c9fe9a6bf73 (127.6.61.2:43655), 105f224cc0d6477d9220d8a5d8c60bb1 (127.6.61.1:33603)
I20260812 07:08:46.126082 27709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:46.118597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:53679 (local address 127.6.61.2:43655)
0812 07:08:46.118790 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:46.118794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:46.118848 (+ 54us) server_negotiation.cc:407] Connection header received
0812 07:08:46.119270 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:46.119279 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:46.119427 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:46.119721 (+ 294us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:08:46.120544 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:46.121511 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:46.124131 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:46.125015 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:46.125187 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:46.125298 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:46.125624 (+ 326us) server_negotiation.cc:299] Negotiation successful
0812 07:08:46.125766 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 07:08:46.126112 27699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:46.118376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:43655 (local address 127.6.61.3:53679)
0812 07:08:46.118746 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:46.118777 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:46.118942 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:46.120049 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:46.120055 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:46.120088 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:08:46.120326 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:46.120341 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:46.121675 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:46.121681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:46.123919 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:46.123951 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:46.125217 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:46.125222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:46.125429 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:46.125455 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 07:08:46.125577 (+ 122us) client_negotiation.cc:240] Negotiation successful
0812 07:08:46.125743 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":1}
I20260812 07:08:46.126493 27715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:46.119856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:41489 (local address 127.6.61.1:33603)
0812 07:08:46.120103 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:46.120108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:46.120129 (+ 21us) server_negotiation.cc:407] Connection header received
0812 07:08:46.120203 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:46.120207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:46.120300 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:46.120421 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:08:46.121166 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:46.122294 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:46.124652 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:46.125481 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:46.125639 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:46.125760 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:46.126080 (+ 320us) server_negotiation.cc:299] Negotiation successful
0812 07:08:46.126221 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260812 07:08:46.127239 27502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20920e918fc54b4c9856d71bfb474613" candidate_uuid: "1072d373e945448d96fb315bca7ed76a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eef84b63f07c492b89734c9fe9a6bf73" is_pre_election: true
I20260812 07:08:46.126715 27728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:46.119104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:33603 (local address 127.6.61.3:41489)
0812 07:08:46.119866 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:46.119890 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:46.120032 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:46.120469 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:46.120474 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:46.120520 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:08:46.120981 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:46.121010 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:46.122506 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:46.122514 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:46.124438 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:46.124449 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:46.125669 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:46.125675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:46.125901 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:46.125937 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 07:08:46.126047 (+ 110us) client_negotiation.cc:240] Negotiation successful
0812 07:08:46.126185 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":30,"thread_start_us":1617,"threads_started":1}
I20260812 07:08:46.127827 27502 raft_consensus.cc:2454] T 20920e918fc54b4c9856d71bfb474613 P eef84b63f07c492b89734c9fe9a6bf73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1072d373e945448d96fb315bca7ed76a in term 0.
I20260812 07:08:46.128711 27570 leader_election.cc:302] T 20920e918fc54b4c9856d71bfb474613 P 1072d373e945448d96fb315bca7ed76a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1072d373e945448d96fb315bca7ed76a, eef84b63f07c492b89734c9fe9a6bf73; no voters:
I20260812 07:08:46.128826 27354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20920e918fc54b4c9856d71bfb474613" candidate_uuid: "1072d373e945448d96fb315bca7ed76a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "105f224cc0d6477d9220d8a5d8c60bb1" is_pre_election: true
I20260812 07:08:46.129181 27725 raft_consensus.cc:2781] T 20920e918fc54b4c9856d71bfb474613 P 1072d373e945448d96fb315bca7ed76a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:08:46.129411 27725 raft_consensus.cc:482] T 20920e918fc54b4c9856d71bfb474613 P 1072d373e945448d96fb315bca7ed76a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:08:46.129441 27354 raft_consensus.cc:2454] T 20920e918fc54b4c9856d71bfb474613 P 105f224cc0d6477d9220d8a5d8c60bb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1072d373e945448d96fb315bca7ed76a in term 0.
I20260812 07:08:46.129516 27725 raft_consensus.cc:3037] T 20920e918fc54b4c9856d71bfb474613 P 1072d373e945448d96fb315bca7ed76a [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:08:46.132004 27725 raft_consensus.cc:504] T 20920e918fc54b4c9856d71bfb474613 P 1072d373e945448d96fb315bca7ed76a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef84b63f07c492b89734c9fe9a6bf73" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 43655 } } peers { permanent_uuid: "1072d373e945448d96fb315bca7ed76a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33833 } } peers { permanent_uuid: "105f224cc0d6477d9220d8a5d8c60bb1" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 33603 } }
I20260812 07:08:46.132733 27725 leader_election.cc:288] T 20920e918fc54b4c9856d71bfb474613 P 1072d373e945448d96fb315bca7ed76a [CANDIDATE]: Term 1 election: Requested vote from peers eef84b63f07c492b89734c9fe9a6bf73 (127.6.61.2:43655), 105f224cc0d6477d9220d8a5d8c60bb1 (127.6.61.1:33603)
I20260812 07:08:46.133471 27502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20920e918fc54b4c9856d71bfb474613" candidate_uuid: "1072d373e945448d96fb315bca7ed76a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eef84b63f07c492b89734c9fe9a6bf73"
I20260812 07:08:46.133672 27354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20920e918fc54b4c9856d71bfb474613" candidate_uuid: "1072d373e945448d96fb315bca7ed76a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "105f224cc0d6477d9220d8a5d8c60bb1"
I20260812 07:08:46.133693 27502 raft_consensus.cc:3037] T 20920e918fc54b4c9856d71bfb474613 P eef84b63f07c492b89734c9fe9a6bf73 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:08:46.133968 27354 raft_consensus.cc:3037] T 20920e918fc54b4c9856d71bfb474613 P 105f224cc0d6477d9220d8a5d8c60bb1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:08:46.136526 27502 raft_consensus.cc:2454] T 20920e918fc54b4c9856d71bfb474613 P eef84b63f07c492b89734c9fe9a6bf73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1072d373e945448d96fb315bca7ed76a in term 1.
I20260812 07:08:46.136580 27354 raft_consensus.cc:2454] T 20920e918fc54b4c9856d71bfb474613 P 105f224cc0d6477d9220d8a5d8c60bb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1072d373e945448d96fb315bca7ed76a in term 1.
I20260812 07:08:46.137162 27570 leader_election.cc:302] T 20920e918fc54b4c9856d71bfb474613 P 1072d373e945448d96fb315bca7ed76a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1072d373e945448d96fb315bca7ed76a, eef84b63f07c492b89734c9fe9a6bf73; no voters:
I20260812 07:08:46.137576 27725 raft_consensus.cc:2781] T 20920e918fc54b4c9856d71bfb474613 P 1072d373e945448d96fb315bca7ed76a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:08:46.138248 27725 raft_consensus.cc:686] T 20920e918fc54b4c9856d71bfb474613 P 1072d373e945448d96fb315bca7ed76a [term 1 LEADER]: Becoming Leader. State: Replica: 1072d373e945448d96fb315bca7ed76a, State: Running, Role: LEADER
I20260812 07:08:46.138783 27725 consensus_queue.cc:237] T 20920e918fc54b4c9856d71bfb474613 P 1072d373e945448d96fb315bca7ed76a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef84b63f07c492b89734c9fe9a6bf73" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 43655 } } peers { permanent_uuid: "1072d373e945448d96fb315bca7ed76a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33833 } } peers { permanent_uuid: "105f224cc0d6477d9220d8a5d8c60bb1" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 33603 } }
I20260812 07:08:46.144476 27194 catalog_manager.cc:5495] T 20920e918fc54b4c9856d71bfb474613 P 1072d373e945448d96fb315bca7ed76a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1072d373e945448d96fb315bca7ed76a (127.6.61.3). New cstate: current_term: 1 leader_uuid: "1072d373e945448d96fb315bca7ed76a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eef84b63f07c492b89734c9fe9a6bf73" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 43655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1072d373e945448d96fb315bca7ed76a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 33833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "105f224cc0d6477d9220d8a5d8c60bb1" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 33603 } health_report { overall_health: UNKNOWN } } }
I20260812 07:08:46.180554 27710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:46.174542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56082 (local address 127.6.61.3:33833)
0812 07:08:46.174753 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:46.174761 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:46.174782 (+ 21us) server_negotiation.cc:407] Connection header received
0812 07:08:46.175057 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:46.175062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:46.175146 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:46.175326 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:08:46.175739 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:46.176507 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:46.178136 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:46.178611 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:46.178760 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:46.179336 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:08:46.179882 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:08:46.180063 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:46.180177 (+ 114us) server_negotiation.cc:299] Negotiation successful
0812 07:08:46.180269 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 07:08:46.190441 27354 raft_consensus.cc:1261] T 20920e918fc54b4c9856d71bfb474613 P 105f224cc0d6477d9220d8a5d8c60bb1 [term 1 FOLLOWER]: Refusing update from remote peer 1072d373e945448d96fb315bca7ed76a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:08:46.190457 27502 raft_consensus.cc:1261] T 20920e918fc54b4c9856d71bfb474613 P eef84b63f07c492b89734c9fe9a6bf73 [term 1 FOLLOWER]: Refusing update from remote peer 1072d373e945448d96fb315bca7ed76a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:08:46.191524 27729 consensus_queue.cc:1035] T 20920e918fc54b4c9856d71bfb474613 P 1072d373e945448d96fb315bca7ed76a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eef84b63f07c492b89734c9fe9a6bf73" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 43655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:08:46.191988 27725 consensus_queue.cc:1035] T 20920e918fc54b4c9856d71bfb474613 P 1072d373e945448d96fb315bca7ed76a [LEADER]: Connected to new peer: Peer: permanent_uuid: "105f224cc0d6477d9220d8a5d8c60bb1" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 33603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 07:08:46.192085 27551 tablet.cc:2369] T 20920e918fc54b4c9856d71bfb474613 P eef84b63f07c492b89734c9fe9a6bf73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:08:46.204111 27734 mvcc.cc:205] Tried to move back new op lower bound from 7317579883263721472 to 7317579883069042688. Current Snapshot: MvccSnapshot[applied={T|T < 7317579883263721472}]
I20260812 07:08:46.205499 27736 mvcc.cc:205] Tried to move back new op lower bound from 7317579883263721472 to 7317579883069042688. Current Snapshot: MvccSnapshot[applied={T|T < 7317579883263721472}]
I20260812 07:08:46.224402 27194 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:08:46.224740 27194 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:08:46.229441 27194 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c666142015d0467080a07128d22ef6c2: Sending DeleteTablet for 3 replicas of tablet 20920e918fc54b4c9856d71bfb474613
I20260812 07:08:46.231382 27480 tablet_service.cc:1514] Processing DeleteTablet for tablet 20920e918fc54b4c9856d71bfb474613 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:08:46 UTC) from {username='slave'} at 127.0.0.1:38546
I20260812 07:08:46.231662 27332 tablet_service.cc:1514] Processing DeleteTablet for tablet 20920e918fc54b4c9856d71bfb474613 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:08:46 UTC) from {username='slave'} at 127.0.0.1:36694
I20260812 07:08:46.232273 27624 tablet_service.cc:1514] Processing DeleteTablet for tablet 20920e918fc54b4c9856d71bfb474613 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:08:46 UTC) from {username='slave'} at 127.0.0.1:56078
I20260812 07:08:46.232542 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 27264
I20260812 07:08:46.250516 27748 tablet_replica.cc:330] T 20920e918fc54b4c9856d71bfb474613 P 1072d373e945448d96fb315bca7ed76a: stopping tablet replica
I20260812 07:08:46.251246 27748 raft_consensus.cc:2229] T 20920e918fc54b4c9856d71bfb474613 P 1072d373e945448d96fb315bca7ed76a [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:08:46.252285 27748 raft_consensus.cc:2258] T 20920e918fc54b4c9856d71bfb474613 P 1072d373e945448d96fb315bca7ed76a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:08:46.253166 27745 tablet_replica.cc:330] T 20920e918fc54b4c9856d71bfb474613 P eef84b63f07c492b89734c9fe9a6bf73: stopping tablet replica
I20260812 07:08:46.253190 27747 tablet_replica.cc:330] T 20920e918fc54b4c9856d71bfb474613 P 105f224cc0d6477d9220d8a5d8c60bb1: stopping tablet replica
I20260812 07:08:46.253659 27745 raft_consensus.cc:2229] T 20920e918fc54b4c9856d71bfb474613 P eef84b63f07c492b89734c9fe9a6bf73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:08:46.253659 27747 raft_consensus.cc:2229] T 20920e918fc54b4c9856d71bfb474613 P 105f224cc0d6477d9220d8a5d8c60bb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:08:46.254211 27745 raft_consensus.cc:2258] T 20920e918fc54b4c9856d71bfb474613 P eef84b63f07c492b89734c9fe9a6bf73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:08:46.255189 27748 ts_tablet_manager.cc:1905] T 20920e918fc54b4c9856d71bfb474613 P 1072d373e945448d96fb315bca7ed76a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:08:46.256487 27747 raft_consensus.cc:2258] T 20920e918fc54b4c9856d71bfb474613 P 105f224cc0d6477d9220d8a5d8c60bb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:08:46.256886 27745 ts_tablet_manager.cc:1905] T 20920e918fc54b4c9856d71bfb474613 P eef84b63f07c492b89734c9fe9a6bf73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:08:46.259059 27715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:46.234087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36700 (local address 127.6.61.1:33603)
0812 07:08:46.234320 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:46.234327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:46.243817 (+ 9490us) server_negotiation.cc:407] Connection header received
0812 07:08:46.244451 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:46.244461 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:46.244610 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:46.244720 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:46.245682 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:46.247282 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:46.253986 (+ 6704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:46.254689 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:46.254781 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:46.257280 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:46.257292 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:46.257309 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:08:46.258116 (+ 807us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:46.258181 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:08:46.258216 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:46.258307 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:46.258570 (+ 263us) server_negotiation.cc:299] Negotiation successful
0812 07:08:46.258702 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260812 07:08:46.260327 27400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:08:46.264484 27748 ts_tablet_manager.cc:1918] T 20920e918fc54b4c9856d71bfb474613 P 1072d373e945448d96fb315bca7ed76a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:08:46.264747 27748 log.cc:1196] T 20920e918fc54b4c9856d71bfb474613 P 1072d373e945448d96fb315bca7ed76a: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-2/wal/wals/20920e918fc54b4c9856d71bfb474613
I20260812 07:08:46.265453 27748 ts_tablet_manager.cc:1939] T 20920e918fc54b4c9856d71bfb474613 P 1072d373e945448d96fb315bca7ed76a: Deleting consensus metadata
I20260812 07:08:46.265988 27745 ts_tablet_manager.cc:1918] T 20920e918fc54b4c9856d71bfb474613 P eef84b63f07c492b89734c9fe9a6bf73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:08:46.266212 27745 log.cc:1196] T 20920e918fc54b4c9856d71bfb474613 P eef84b63f07c492b89734c9fe9a6bf73: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518208686283-6388-0/minicluster-data/ts-1/wal/wals/20920e918fc54b4c9856d71bfb474613
I20260812 07:08:46.266894 27745 ts_tablet_manager.cc:1939] T 20920e918fc54b4c9856d71bfb474613 P eef84b63f07c492b89734c9fe9a6bf73: Deleting consensus metadata
I20260812 07:08:46.267782 27178 catalog_manager.cc:4841] TS 1072d373e945448d96fb315bca7ed76a (127.6.61.3:33833): tablet 20920e918fc54b4c9856d71bfb474613 (table test-table [id=86c87d191b3b4e78887976a604430eb2]) successfully deleted
I20260812 07:08:46.269474 27179 catalog_manager.cc:4841] TS eef84b63f07c492b89734c9fe9a6bf73 (127.6.61.2:43655): tablet 20920e918fc54b4c9856d71bfb474613 (table test-table [id=86c87d191b3b4e78887976a604430eb2]) successfully deleted
I20260812 07:08:46.360903 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 27264
I20260812 07:08:46.389734 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 27409
W20260812 07:08:46.390209 27179 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.61.1:33603 (error 108)
I20260812 07:08:46.402271 27709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:46.390973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38552 (local address 127.6.61.2:43655)
0812 07:08:46.391182 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:46.391191 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:46.391213 (+ 22us) server_negotiation.cc:407] Connection header received
0812 07:08:46.391300 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:46.391308 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:46.391481 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:46.391623 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:46.392441 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:46.393976 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:46.396607 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:46.397505 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:46.397673 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:46.399688 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:46.399704 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:46.399731 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:08:46.400744 (+ 1013us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:46.400840 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:08:46.400876 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:46.401002 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:46.401712 (+ 710us) server_negotiation.cc:299] Negotiation successful
0812 07:08:46.401863 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 07:08:46.403004 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:46.402457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:33603 (local address 127.0.0.1:36712)
0812 07:08:46.402679 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:46.402834 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:33603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260812 07:08:46.403359 27179 catalog_manager.cc:4584] TS 105f224cc0d6477d9220d8a5d8c60bb1 (127.6.61.1:33603): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 20920e918fc54b4c9856d71bfb474613: Network error: Client connection negotiation failed: client connection to 127.6.61.1:33603: connect: Connection refused (error 111)
I20260812 07:08:46.403519 27544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:08:46.457543 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:46.456910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:33603 (local address 127.0.0.1:36726)
0812 07:08:46.457200 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:46.457318 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:33603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260812 07:08:46.458911 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:46.458453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:33603 (local address 127.0.0.1:36736)
0812 07:08:46.458642 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:46.458733 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:33603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260812 07:08:46.495023 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 27409
I20260812 07:08:46.519515 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:46.519030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:33603 (local address 127.0.0.1:36746)
0812 07:08:46.519248 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:46.519338 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:33603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260812 07:08:46.520663 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:46.520268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:33603 (local address 127.0.0.1:36758)
0812 07:08:46.520468 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:46.520539 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:33603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260812 07:08:46.523270 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 27553
I20260812 07:08:46.531224 27710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:46.524268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56088 (local address 127.6.61.3:33833)
0812 07:08:46.524486 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:46.524494 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:46.524516 (+ 22us) server_negotiation.cc:407] Connection header received
0812 07:08:46.524586 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:46.524590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:46.524682 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:46.524808 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:46.525254 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:46.526202 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:46.527575 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:46.528089 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:46.528237 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:46.529493 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:46.529504 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:46.529521 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:08:46.530348 (+ 827us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:46.530406 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:08:46.530429 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:46.530562 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:46.530719 (+ 157us) server_negotiation.cc:299] Negotiation successful
0812 07:08:46.530838 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 07:08:46.532027 27691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:08:46.601434 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:46.600957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:33603 (local address 127.0.0.1:36762)
0812 07:08:46.601177 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:46.601272 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:33603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260812 07:08:46.602595 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:46.602162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:33603 (local address 127.0.0.1:36764)
0812 07:08:46.602405 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:46.602484 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:33603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260812 07:08:46.656998 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 27553
I20260812 07:08:46.684767 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 27163
I20260812 07:08:46.686076 27232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:08:46.801631 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:46.800821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:33603 (local address 127.0.0.1:36768)
0812 07:08:46.801362 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:46.801463 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:33603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350}
I20260812 07:08:46.801687 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 27163
2026-08-12T07:08:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3010 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:08:46 dist-test-slave-69rg krb5kdc[27757](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:08:46 dist-test-slave-69rg krb5kdc[27757](info): set up 2 sockets
Aug 12 07:08:46 dist-test-slave-69rg krb5kdc[27757](info): commencing operation
krb5kdc: starting...
W20260812 07:08:48.069885 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.225s user 0.001s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:08:48 dist-test-slave-69rg krb5kdc[27757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:08:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:08:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
I20260812 07:08:48.402298 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:44631
--webserver_interface=127.6.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:43281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:44631
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:08:48.764015 27772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:08:48.764420 27772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:08:48.764599 27772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:08:48.772219 27772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:08:48.772399 27772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:08:48.772518 27772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:08:48.772641 27772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:08:48.772722 27772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:08:48.785419 27772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:43281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:44631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:44631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:08:48.787003 27772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:08:48.788445 27772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:08:48.801644 27778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:48.801729 27779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:48.801705 27781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:08:48.802660 27772 server_base.cc:1034] running on GCE node
I20260812 07:08:48.803555 27772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:08:48.804961 27772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:08:48.806213 27772 hybrid_clock.cc:648] HybridClock initialized: now 1786518528806167 us; error 51 us; skew 500 ppm
I20260812 07:08:48.806608 27772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:08:48 dist-test-slave-69rg krb5kdc[27757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518528, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:08:48.810082 27772 init.cc:377] Logged in from keytab as kudu/127.6.61.62@KRBTEST.COM (short username kudu)
I20260812 07:08:48.811712 27772 webserver.cc:460] Webserver started at http://127.6.61.62:46505/ using document root <none> and password file <none>
I20260812 07:08:48.812433 27772 fs_manager.cc:359] Metadata directory not provided
I20260812 07:08:48.812592 27772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:08:48.812930 27772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:08:48.816123 27772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "82396e94dc00454c83436b77bd11c091"
format_stamp: "Formatted at 2026-08-12 07:08:48 on dist-test-slave-69rg"
server_key: "870e62f504fa98f372293b7ac660d695"
server_key_iv: "8fd5a1c7817be2b23c8784e3247924f7"
server_key_version: "encryptionkey@0"
I20260812 07:08:48.816934 27772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "82396e94dc00454c83436b77bd11c091"
format_stamp: "Formatted at 2026-08-12 07:08:48 on dist-test-slave-69rg"
server_key: "870e62f504fa98f372293b7ac660d695"
server_key_iv: "8fd5a1c7817be2b23c8784e3247924f7"
server_key_version: "encryptionkey@0"
I20260812 07:08:48.822477 27772 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 07:08:48.826118 27788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:48.827514 27772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:08:48.827744 27772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "82396e94dc00454c83436b77bd11c091"
format_stamp: "Formatted at 2026-08-12 07:08:48 on dist-test-slave-69rg"
server_key: "870e62f504fa98f372293b7ac660d695"
server_key_iv: "8fd5a1c7817be2b23c8784e3247924f7"
server_key_version: "encryptionkey@0"
I20260812 07:08:48.827975 27772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:08:48.861205 27772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:08:48.866078 27772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:08:48.866420 27772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:08:48.887056 27772 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:44631
I20260812 07:08:48.887116 27848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:44631 every 8 connection(s)
I20260812 07:08:48.888712 27772 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:08:48.892771 27849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:08:48.896517 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 27772
I20260812 07:08:48.896751 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:08:48.897331 6388 external_mini_cluster.cc:1428] Setting key ad2448df2ed0b2d958031150ec4afcbf
I20260812 07:08:48.906713 27849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82396e94dc00454c83436b77bd11c091: Bootstrap starting.
I20260812 07:08:48.910710 27849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82396e94dc00454c83436b77bd11c091: Neither blocks nor log segments found. Creating new log.
I20260812 07:08:48.912132 27849 log.cc:824] T 00000000000000000000000000000000 P 82396e94dc00454c83436b77bd11c091: Log is configured to *not* fsync() on all Append() calls
Aug 12 07:08:48 dist-test-slave-69rg krb5kdc[27757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:08:48.915832 27849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82396e94dc00454c83436b77bd11c091: No bootstrap required, opened a new log
I20260812 07:08:48.923241 27852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:48.899496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37928 (local address 127.6.61.62:44631)
0812 07:08:48.900302 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:48.900314 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:48.900361 (+ 47us) server_negotiation.cc:407] Connection header received
0812 07:08:48.902145 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:48.902171 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:48.902610 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:48.902958 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:48.905035 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:48.906494 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:48.908367 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:48.909077 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:48.909229 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:48.915031 (+ 5802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:48.915039 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:48.915054 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:08:48.915095 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:48.918655 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:48.919706 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:48.919714 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:48.919723 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:48.919934 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:48.920374 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:48.920378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:48.920380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:48.920933 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:48.921227 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:48.921657 (+ 430us) server_negotiation.cc:299] Negotiation successful
0812 07:08:48.922177 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":378,"threads_started":1}
I20260812 07:08:48.925246 27849 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82396e94dc00454c83436b77bd11c091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82396e94dc00454c83436b77bd11c091" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 44631 } }
I20260812 07:08:48.925727 27849 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82396e94dc00454c83436b77bd11c091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:08:48.925876 27849 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82396e94dc00454c83436b77bd11c091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82396e94dc00454c83436b77bd11c091, State: Initialized, Role: FOLLOWER
I20260812 07:08:48.926533 27849 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82396e94dc00454c83436b77bd11c091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82396e94dc00454c83436b77bd11c091" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 44631 } }
I20260812 07:08:48.926789 27849 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82396e94dc00454c83436b77bd11c091 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:08:48.926891 27849 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82396e94dc00454c83436b77bd11c091 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:08:48.927013 27849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82396e94dc00454c83436b77bd11c091 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:08:48.929106 27849 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82396e94dc00454c83436b77bd11c091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82396e94dc00454c83436b77bd11c091" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 44631 } }
I20260812 07:08:48.929543 27849 leader_election.cc:302] T 00000000000000000000000000000000 P 82396e94dc00454c83436b77bd11c091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82396e94dc00454c83436b77bd11c091; no voters:
I20260812 07:08:48.930141 27849 leader_election.cc:288] T 00000000000000000000000000000000 P 82396e94dc00454c83436b77bd11c091 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:08:48.930277 27855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82396e94dc00454c83436b77bd11c091 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:08:48.931082 27855 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82396e94dc00454c83436b77bd11c091 [term 1 LEADER]: Becoming Leader. State: Replica: 82396e94dc00454c83436b77bd11c091, State: Running, Role: LEADER
I20260812 07:08:48.931598 27855 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82396e94dc00454c83436b77bd11c091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82396e94dc00454c83436b77bd11c091" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 44631 } }
I20260812 07:08:48.932546 27849 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82396e94dc00454c83436b77bd11c091 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:08:48.935022 27857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82396e94dc00454c83436b77bd11c091 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82396e94dc00454c83436b77bd11c091. Latest consensus state: current_term: 1 leader_uuid: "82396e94dc00454c83436b77bd11c091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82396e94dc00454c83436b77bd11c091" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 44631 } } }
I20260812 07:08:48.934998 27856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82396e94dc00454c83436b77bd11c091 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82396e94dc00454c83436b77bd11c091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82396e94dc00454c83436b77bd11c091" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 44631 } } }
I20260812 07:08:48.935504 27856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82396e94dc00454c83436b77bd11c091 [sys.catalog]: This master's current role is: LEADER
I20260812 07:08:48.935478 27857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82396e94dc00454c83436b77bd11c091 [sys.catalog]: This master's current role is: LEADER
I20260812 07:08:48.939286 27861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:08:48.947009 27861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:08:48.956631 27861 catalog_manager.cc:1347] Generated new cluster ID: a7e3854586194dd580f06816576de643
I20260812 07:08:48.956776 27861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:08:48.973609 27861 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:08:48.975419 27861 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:08:48.982287 27861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82396e94dc00454c83436b77bd11c091: Generated new TSK 0
I20260812 07:08:48.983279 27861 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
I20260812 07:08:49.074195 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.1:0
--local_ip_for_outbound_sockets=127.6.61.1
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:44631
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:43281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:08:49.429239 27877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:08:49.429524 27877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:08:49.429689 27877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:08:49.437595 27877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:08:49.437707 27877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:08:49.437928 27877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.1
I20260812 07:08:49.448140 27877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:43281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:44631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.1
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:08:49.449512 27877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:08:49.450932 27877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:08:49.465083 27882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:49.465281 27883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:49.465216 27885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:08:49.467626 27877 server_base.cc:1034] running on GCE node
I20260812 07:08:49.468369 27877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:08:49.469424 27877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:08:49.470678 27877 hybrid_clock.cc:648] HybridClock initialized: now 1786518529470639 us; error 29 us; skew 500 ppm
I20260812 07:08:49.471060 27877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:08:49 dist-test-slave-69rg krb5kdc[27757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518529, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:08:49.474560 27877 init.cc:377] Logged in from keytab as kudu/127.6.61.1@KRBTEST.COM (short username kudu)
I20260812 07:08:49.476215 27877 webserver.cc:460] Webserver started at http://127.6.61.1:45015/ using document root <none> and password file <none>
I20260812 07:08:49.476958 27877 fs_manager.cc:359] Metadata directory not provided
I20260812 07:08:49.477116 27877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:08:49.477444 27877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:08:49.480489 27877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-0/data/instance:
uuid: "cfa8a45cc47d4e97a3496e30c4d13366"
format_stamp: "Formatted at 2026-08-12 07:08:49 on dist-test-slave-69rg"
server_key: "355724163d401dc8e2423440fd966c23"
server_key_iv: "e3e62fe6270b4afffacf02d87cb07948"
server_key_version: "encryptionkey@0"
I20260812 07:08:49.481325 27877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance:
uuid: "cfa8a45cc47d4e97a3496e30c4d13366"
format_stamp: "Formatted at 2026-08-12 07:08:49 on dist-test-slave-69rg"
server_key: "355724163d401dc8e2423440fd966c23"
server_key_iv: "e3e62fe6270b4afffacf02d87cb07948"
server_key_version: "encryptionkey@0"
I20260812 07:08:49.486936 27877 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260812 07:08:49.490572 27893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:49.492000 27877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:08:49.492194 27877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-0/wal
uuid: "cfa8a45cc47d4e97a3496e30c4d13366"
format_stamp: "Formatted at 2026-08-12 07:08:49 on dist-test-slave-69rg"
server_key: "355724163d401dc8e2423440fd966c23"
server_key_iv: "e3e62fe6270b4afffacf02d87cb07948"
server_key_version: "encryptionkey@0"
I20260812 07:08:49.492419 27877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:08:49.528661 27877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:08:49.533464 27877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:08:49.533776 27877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:08:49.534894 27877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:08:49.536324 27877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:08:49.536432 27877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:49.536599 27877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:08:49.536675 27877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:49.572386 27877 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.1:33025
I20260812 07:08:49.572479 28016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.1:33025 every 8 connection(s)
I20260812 07:08:49.573776 27877 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
I20260812 07:08:49.579445 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 27877
I20260812 07:08:49.579648 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance
I20260812 07:08:49.580209 6388 external_mini_cluster.cc:1428] Setting key 1f7d0e3c176a37e2c8681e6ad7bc4609
Aug 12 07:08:49 dist-test-slave-69rg krb5kdc[27757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518529, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:08:49.598323 28019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:49.576698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:42713 (local address 127.6.61.62:44631)
0812 07:08:49.577182 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:49.577187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:49.578178 (+ 991us) server_negotiation.cc:407] Connection header received
0812 07:08:49.580694 (+ 2516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:49.580699 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:49.580772 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:49.580928 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:49.582805 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:49.583532 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:49.586163 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:49.586574 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:49.586698 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:49.591386 (+ 4688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:49.591392 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:49.591395 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:08:49.591433 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:49.594527 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:49.595436 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:49.595442 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:49.595444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:49.595575 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:49.596110 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:49.596115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:49.596117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:49.596427 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:49.596642 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:49.597594 (+ 952us) server_negotiation.cc:299] Negotiation successful
0812 07:08:49.597880 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":233,"threads_started":1}
I20260812 07:08:49.599673 28020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:49.577221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:44631 (local address 127.6.61.1:42713)
0812 07:08:49.577997 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:49.578047 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:49.580360 (+ 2313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:49.581185 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:49.581197 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:49.581777 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:08:49.582614 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:49.582641 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:49.583715 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:49.583723 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:49.585995 (+ 2272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:49.586014 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:49.586847 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:49.586856 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:49.587188 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:49.587988 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:08:49.588030 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:08:49.591094 (+ 3064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:08:49.594708 (+ 3614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:08:49.594719 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:08:49.594741 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:08:49.595252 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:08:49.595707 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:08:49.595713 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:08:49.595717 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:08:49.595965 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:08:49.596649 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:08:49.596660 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:08:49.597164 (+ 504us) client_negotiation.cc:769] Sending connection context
0812 07:08:49.597464 (+ 300us) client_negotiation.cc:240] Negotiation successful
0812 07:08:49.598137 (+ 673us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":291,"threads_started":1}
I20260812 07:08:49.601511 28017 heartbeater.cc:347] Connected to a master server at 127.6.61.62:44631
I20260812 07:08:49.601902 28017 heartbeater.cc:464] Registering TS with master...
I20260812 07:08:49.602893 28017 heartbeater.cc:511] Master 127.6.61.62:44631 requested a full tablet report, sending...
I20260812 07:08:49.605893 27806 ts_manager.cc:194] Registered new tserver with Master: cfa8a45cc47d4e97a3496e30c4d13366 (127.6.61.1:33025)
I20260812 07:08:49.608323 27806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.1@KRBTEST.COM'} at 127.6.61.1:42713
WARNING: no policy specified for kudu/127.6.61.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
I20260812 07:08:49.661649 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.2:0
--local_ip_for_outbound_sockets=127.6.61.2
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:44631
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:43281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:08:50.014098 28024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:08:50.014454 28024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:08:50.014633 28024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:08:50.022120 28024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:08:50.022277 28024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:08:50.022449 28024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.2
I20260812 07:08:50.032574 28024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:43281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:44631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.2
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:08:50.033977 28024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:08:50.035332 28024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:08:50.049641 28033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:50.049726 28030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:50.049677 28029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:08:50.056041 28024 server_base.cc:1034] running on GCE node
I20260812 07:08:50.056886 28024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:08:50.058087 28024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:08:50.059368 28024 hybrid_clock.cc:648] HybridClock initialized: now 1786518530059334 us; error 72 us; skew 500 ppm
I20260812 07:08:50.059837 28024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:08:50 dist-test-slave-69rg krb5kdc[27757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518530, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:08:50.064015 28024 init.cc:377] Logged in from keytab as kudu/127.6.61.2@KRBTEST.COM (short username kudu)
I20260812 07:08:50.071069 28024 webserver.cc:460] Webserver started at http://127.6.61.2:34399/ using document root <none> and password file <none>
I20260812 07:08:50.072038 28024 fs_manager.cc:359] Metadata directory not provided
I20260812 07:08:50.072224 28024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:08:50.072661 28024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:08:50.076596 28024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-1/data/instance:
uuid: "8ab11dc05044446fb867623234cd9a19"
format_stamp: "Formatted at 2026-08-12 07:08:50 on dist-test-slave-69rg"
server_key: "70b1b6cad03b71b1c50fc62bc23858a6"
server_key_iv: "6c48eb036a54c6a4342fd752ccf05921"
server_key_version: "encryptionkey@0"
I20260812 07:08:50.077473 28024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance:
uuid: "8ab11dc05044446fb867623234cd9a19"
format_stamp: "Formatted at 2026-08-12 07:08:50 on dist-test-slave-69rg"
server_key: "70b1b6cad03b71b1c50fc62bc23858a6"
server_key_iv: "6c48eb036a54c6a4342fd752ccf05921"
server_key_version: "encryptionkey@0"
I20260812 07:08:50.083235 28024 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 07:08:50.086905 28040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:50.088490 28024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:08:50.088730 28024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-1/wal
uuid: "8ab11dc05044446fb867623234cd9a19"
format_stamp: "Formatted at 2026-08-12 07:08:50 on dist-test-slave-69rg"
server_key: "70b1b6cad03b71b1c50fc62bc23858a6"
server_key_iv: "6c48eb036a54c6a4342fd752ccf05921"
server_key_version: "encryptionkey@0"
I20260812 07:08:50.088970 28024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:08:50.105782 28024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:08:50.110489 28024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:08:50.110757 28024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:08:50.111721 28024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:08:50.113042 28024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:08:50.113118 28024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:50.113209 28024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:08:50.113255 28024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:50.149078 28024 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.2:43871
I20260812 07:08:50.149199 28164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.2:43871 every 8 connection(s)
I20260812 07:08:50.150652 28024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
I20260812 07:08:50.150962 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 28024
I20260812 07:08:50.151162 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance
I20260812 07:08:50.151840 6388 external_mini_cluster.cc:1428] Setting key 5a9b9ce0fa115b9bef25ec01e812728c
Aug 12 07:08:50 dist-test-slave-69rg krb5kdc[27757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518530, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:08:50.173520 28167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:50.153762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:59367 (local address 127.6.61.62:44631)
0812 07:08:50.154338 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:50.154345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:50.155524 (+ 1179us) server_negotiation.cc:407] Connection header received
0812 07:08:50.157611 (+ 2087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:50.157616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:50.157705 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:50.157862 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:50.159508 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:50.160283 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:50.161975 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:50.162431 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:50.162577 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:50.166385 (+ 3808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:50.166391 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:50.166394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:08:50.166431 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:50.169765 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:50.170923 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:50.170927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:50.170930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:50.171074 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:50.171616 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:50.171619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:50.171621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:50.171920 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:50.172038 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:50.173026 (+ 988us) server_negotiation.cc:299] Negotiation successful
0812 07:08:50.173217 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":284,"threads_started":1}
I20260812 07:08:50.175129 28168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:50.154470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:44631 (local address 127.6.61.2:59367)
0812 07:08:50.155338 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:50.155378 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:50.157208 (+ 1830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:50.158134 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:50.158143 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:50.158724 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:08:50.159322 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:50.159340 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:50.160462 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:50.160467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:50.161767 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:50.161780 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:50.162689 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:50.162698 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:50.162953 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:50.163720 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:08:50.163745 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:08:50.166113 (+ 2368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:08:50.170005 (+ 3892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:08:50.170017 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:08:50.170040 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:08:50.170746 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:08:50.171196 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:08:50.171204 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:08:50.171210 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:08:50.171488 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:08:50.172064 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:08:50.172075 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:08:50.172563 (+ 488us) client_negotiation.cc:769] Sending connection context
0812 07:08:50.172912 (+ 349us) client_negotiation.cc:240] Negotiation successful
0812 07:08:50.173564 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":349,"threads_started":1}
I20260812 07:08:50.176895 28165 heartbeater.cc:347] Connected to a master server at 127.6.61.62:44631
I20260812 07:08:50.177244 28165 heartbeater.cc:464] Registering TS with master...
I20260812 07:08:50.178120 28165 heartbeater.cc:511] Master 127.6.61.62:44631 requested a full tablet report, sending...
I20260812 07:08:50.179845 27806 ts_manager.cc:194] Registered new tserver with Master: 8ab11dc05044446fb867623234cd9a19 (127.6.61.2:43871)
I20260812 07:08:50.181125 27806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.2@KRBTEST.COM'} at 127.6.61.2:59367
WARNING: no policy specified for kudu/127.6.61.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
I20260812 07:08:50.230789 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.3:0
--local_ip_for_outbound_sockets=127.6.61.3
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:44631
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:43281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:08:50.594769 28173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:08:50.595072 28173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:08:50.595166 28173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:08:50.603014 28173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:08:50.603148 28173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:08:50.603375 28173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.3
I20260812 07:08:50.612861 28017 heartbeater.cc:503] Master 127.6.61.62:44631 was elected leader, sending a full tablet report...
I20260812 07:08:50.614074 28173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:43281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:44631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.3
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:08:50.615551 28173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:08:50.616878 28173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:08:50.631023 28178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:50.631750 28179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:50.631827 28181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:08:50.632085 28173 server_base.cc:1034] running on GCE node
I20260812 07:08:50.632887 28173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:08:50.634035 28173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:08:50.635308 28173 hybrid_clock.cc:648] HybridClock initialized: now 1786518530635205 us; error 85 us; skew 500 ppm
I20260812 07:08:50.635702 28173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:08:50 dist-test-slave-69rg krb5kdc[27757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518530, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:08:50.639375 28173 init.cc:377] Logged in from keytab as kudu/127.6.61.3@KRBTEST.COM (short username kudu)
I20260812 07:08:50.641153 28173 webserver.cc:460] Webserver started at http://127.6.61.3:39673/ using document root <none> and password file <none>
I20260812 07:08:50.641986 28173 fs_manager.cc:359] Metadata directory not provided
I20260812 07:08:50.642140 28173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:08:50.642482 28173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:08:50.645807 28173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-2/data/instance:
uuid: "ad1b955a73c14e3fbf776cdb90e1f74a"
format_stamp: "Formatted at 2026-08-12 07:08:50 on dist-test-slave-69rg"
server_key: "5f1be8de0a60bea5833eaf41918b5a85"
server_key_iv: "4f2a2791d0ae9a77fb4e8abb13db649d"
server_key_version: "encryptionkey@0"
I20260812 07:08:50.646783 28173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance:
uuid: "ad1b955a73c14e3fbf776cdb90e1f74a"
format_stamp: "Formatted at 2026-08-12 07:08:50 on dist-test-slave-69rg"
server_key: "5f1be8de0a60bea5833eaf41918b5a85"
server_key_iv: "4f2a2791d0ae9a77fb4e8abb13db649d"
server_key_version: "encryptionkey@0"
I20260812 07:08:50.652376 28173 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 07:08:50.656116 28188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:50.657573 28173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 07:08:50.657785 28173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-2/wal
uuid: "ad1b955a73c14e3fbf776cdb90e1f74a"
format_stamp: "Formatted at 2026-08-12 07:08:50 on dist-test-slave-69rg"
server_key: "5f1be8de0a60bea5833eaf41918b5a85"
server_key_iv: "4f2a2791d0ae9a77fb4e8abb13db649d"
server_key_version: "encryptionkey@0"
I20260812 07:08:50.658052 28173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:08:50.689745 28173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:08:50.694629 28173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:08:50.694960 28173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:08:50.696154 28173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:08:50.697749 28173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:08:50.697872 28173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260812 07:08:50.698043 28173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:08:50.698127 28173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:50.734622 28173 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.3:36359
I20260812 07:08:50.734679 28311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.3:36359 every 8 connection(s)
I20260812 07:08:50.736112 28173 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
I20260812 07:08:50.736488 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 28173
I20260812 07:08:50.736716 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance
I20260812 07:08:50.737371 6388 external_mini_cluster.cc:1428] Setting key 7531c2f4204a948fa914856bbba170af
Aug 12 07:08:50 dist-test-slave-69rg krb5kdc[27757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518530, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:08:50.758694 28314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:50.739162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:53593 (local address 127.6.61.62:44631)
0812 07:08:50.739711 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:50.739719 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:50.741015 (+ 1296us) server_negotiation.cc:407] Connection header received
0812 07:08:50.743005 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:50.743009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:50.743082 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:50.743193 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:50.744758 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:50.745562 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:50.747392 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:50.747846 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:50.747959 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:50.752039 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:50.752046 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:50.752050 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:08:50.752095 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:50.755453 (+ 3358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:50.756265 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:50.756269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:50.756271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:50.756407 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:50.756829 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:50.756834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:50.756836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:50.757121 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:50.757280 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:50.758202 (+ 922us) server_negotiation.cc:299] Negotiation successful
0812 07:08:50.758412 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":260,"threads_started":1}
I20260812 07:08:50.759680 28315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:50.739872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:44631 (local address 127.6.61.3:53593)
0812 07:08:50.740806 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:50.740892 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:50.742707 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:50.743415 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:50.743423 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:50.743989 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:08:50.744586 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:50.744605 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:50.745740 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:50.745744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:50.747221 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:50.747236 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:50.748077 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:50.748083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:50.748323 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:50.749004 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:08:50.749029 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:08:50.751746 (+ 2717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:08:50.755601 (+ 3855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:08:50.755609 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:08:50.755624 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:08:50.756090 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:08:50.756510 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:08:50.756513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:08:50.756516 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:08:50.756705 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:08:50.757312 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:08:50.757318 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:08:50.757666 (+ 348us) client_negotiation.cc:769] Sending connection context
0812 07:08:50.757979 (+ 313us) client_negotiation.cc:240] Negotiation successful
0812 07:08:50.758451 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":415,"threads_started":1}
I20260812 07:08:50.761337 28312 heartbeater.cc:347] Connected to a master server at 127.6.61.62:44631
I20260812 07:08:50.761695 28312 heartbeater.cc:464] Registering TS with master...
I20260812 07:08:50.762558 28312 heartbeater.cc:511] Master 127.6.61.62:44631 requested a full tablet report, sending...
I20260812 07:08:50.764118 27806 ts_manager.cc:194] Registered new tserver with Master: ad1b955a73c14e3fbf776cdb90e1f74a (127.6.61.3:36359)
I20260812 07:08:50.765304 27806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.3@KRBTEST.COM'} at 127.6.61.3:53593
I20260812 07:08:50.766474 6388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 07:08:50.786692 28314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:50.770697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59890 (local address 127.6.61.62:44631)
0812 07:08:50.770906 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:50.770911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:50.771432 (+ 521us) server_negotiation.cc:407] Connection header received
0812 07:08:50.771569 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:50.771573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:50.771653 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:50.771741 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:50.773297 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:50.774183 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:50.775794 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:50.776195 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:50.776304 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:50.779519 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:50.779524 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:50.779527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:08:50.779566 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:50.782692 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:50.784056 (+ 1364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:50.784062 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:50.784068 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:50.784290 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:50.784880 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:50.784885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:50.784889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:50.785269 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:50.785603 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:50.786291 (+ 688us) server_negotiation.cc:299] Negotiation successful
0812 07:08:50.786452 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 07:08:50.791124 27806 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:08:50.799067 27806 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:08:50.837359 28334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:50.814724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55572 (local address 127.6.61.3:36359)
0812 07:08:50.821627 (+ 6903us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:50.821638 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:50.828637 (+ 6999us) server_negotiation.cc:407] Connection header received
0812 07:08:50.828874 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:50.828880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:50.829078 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:50.829288 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:08:50.830147 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:50.831658 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:50.834441 (+ 2783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:50.835858 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:50.836003 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:50.836254 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:50.836699 (+ 445us) server_negotiation.cc:299] Negotiation successful
0812 07:08:50.836932 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6686,"thread_start_us":6480,"threads_started":1}
I20260812 07:08:50.837359 28329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:50.813101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:36359 (local address 127.0.0.1:55572)
0812 07:08:50.828499 (+ 15398us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:50.828520 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:50.828680 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:50.829350 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:50.829356 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:50.829384 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:08:50.829898 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:50.829913 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:50.831817 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:50.831827 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:50.834269 (+ 2442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:50.834284 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:50.836067 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:50.836074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:50.836376 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:50.836446 (+ 70us) client_negotiation.cc:769] Sending connection context
0812 07:08:50.836581 (+ 135us) client_negotiation.cc:240] Negotiation successful
0812 07:08:50.836799 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15267,"mutex_wait_us":7,"thread_start_us":14736,"threads_started":1}
I20260812 07:08:50.837420 28332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:50.814097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52218 (local address 127.6.61.1:33025)
0812 07:08:50.821633 (+ 7536us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:50.821640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:50.826162 (+ 4522us) server_negotiation.cc:407] Connection header received
0812 07:08:50.827544 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:50.827559 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:50.827891 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:50.828119 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:08:50.829241 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:50.831009 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:50.834893 (+ 3884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:50.835887 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:50.836050 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:50.836283 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:50.836847 (+ 564us) server_negotiation.cc:299] Negotiation successful
0812 07:08:50.837004 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7307,"thread_start_us":7386,"threads_started":1}
I20260812 07:08:50.837380 28327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:50.812533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:33025 (local address 127.0.0.1:52218)
0812 07:08:50.825892 (+ 13359us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:50.825950 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:50.827158 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:50.828491 (+ 1333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:50.828498 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:50.828535 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:08:50.829017 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:50.829035 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:50.831152 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:50.831165 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:50.834728 (+ 3563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:50.834743 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:50.836439 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:50.836446 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:50.836619 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:50.836648 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 07:08:50.836743 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 07:08:50.836847 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12963,"mutex_wait_us":72,"spinlock_wait_cycles":2816,"thread_start_us":12768,"threads_started":1}
I20260812 07:08:50.838590 28330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:50.812645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:43871 (local address 127.0.0.1:42446)
0812 07:08:50.828184 (+ 15539us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:50.828205 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:50.828360 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:50.829083 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:50.829088 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:50.829117 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:08:50.829669 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:50.829686 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:50.831658 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:50.831663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:50.834092 (+ 2429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:50.834109 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:50.837711 (+ 3602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:50.837717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:50.837912 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:50.837945 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 07:08:50.838022 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 07:08:50.838174 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15368,"mutex_wait_us":74,"thread_start_us":15580,"threads_started":1}
I20260812 07:08:50.839466 28328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:50.812643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42446 (local address 127.6.61.2:43871)
0812 07:08:50.826703 (+ 14060us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:50.826716 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:50.828485 (+ 1769us) server_negotiation.cc:407] Connection header received
0812 07:08:50.828607 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:50.828616 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:50.828860 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:50.829071 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:08:50.829953 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:50.831512 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:50.834270 (+ 2758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:50.835857 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:50.836004 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:50.836218 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:50.838991 (+ 2773us) server_negotiation.cc:299] Negotiation successful
0812 07:08:50.839195 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13847,"thread_start_us":14098,"threads_started":1}
I20260812 07:08:50.841630 28243 tablet_service.cc:1467] Processing CreateTablet for tablet d4c2d9e21a024df5b9809574fd006d14 (DEFAULT_TABLE table=test-table [id=163d4aa3fb1742f89ed8c70a7a45ee98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:08:50.841770 27943 tablet_service.cc:1467] Processing CreateTablet for tablet d4c2d9e21a024df5b9809574fd006d14 (DEFAULT_TABLE table=test-table [id=163d4aa3fb1742f89ed8c70a7a45ee98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:08:50.843034 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4c2d9e21a024df5b9809574fd006d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:08:50.843058 27943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4c2d9e21a024df5b9809574fd006d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:08:50.848009 28095 tablet_service.cc:1467] Processing CreateTablet for tablet d4c2d9e21a024df5b9809574fd006d14 (DEFAULT_TABLE table=test-table [id=163d4aa3fb1742f89ed8c70a7a45ee98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:08:50.849602 28095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4c2d9e21a024df5b9809574fd006d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:08:50.855577 28335 tablet_bootstrap.cc:492] T d4c2d9e21a024df5b9809574fd006d14 P cfa8a45cc47d4e97a3496e30c4d13366: Bootstrap starting.
I20260812 07:08:50.858525 28336 tablet_bootstrap.cc:492] T d4c2d9e21a024df5b9809574fd006d14 P ad1b955a73c14e3fbf776cdb90e1f74a: Bootstrap starting.
I20260812 07:08:50.859666 28335 tablet_bootstrap.cc:654] T d4c2d9e21a024df5b9809574fd006d14 P cfa8a45cc47d4e97a3496e30c4d13366: Neither blocks nor log segments found. Creating new log.
I20260812 07:08:50.861052 28335 log.cc:824] T d4c2d9e21a024df5b9809574fd006d14 P cfa8a45cc47d4e97a3496e30c4d13366: Log is configured to *not* fsync() on all Append() calls
I20260812 07:08:50.863468 28336 tablet_bootstrap.cc:654] T d4c2d9e21a024df5b9809574fd006d14 P ad1b955a73c14e3fbf776cdb90e1f74a: Neither blocks nor log segments found. Creating new log.
I20260812 07:08:50.864857 28338 tablet_bootstrap.cc:492] T d4c2d9e21a024df5b9809574fd006d14 P 8ab11dc05044446fb867623234cd9a19: Bootstrap starting.
I20260812 07:08:50.864845 28336 log.cc:824] T d4c2d9e21a024df5b9809574fd006d14 P ad1b955a73c14e3fbf776cdb90e1f74a: Log is configured to *not* fsync() on all Append() calls
I20260812 07:08:50.865536 28335 tablet_bootstrap.cc:492] T d4c2d9e21a024df5b9809574fd006d14 P cfa8a45cc47d4e97a3496e30c4d13366: No bootstrap required, opened a new log
I20260812 07:08:50.865928 28335 ts_tablet_manager.cc:1397] T d4c2d9e21a024df5b9809574fd006d14 P cfa8a45cc47d4e97a3496e30c4d13366: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260812 07:08:50.868445 28338 tablet_bootstrap.cc:654] T d4c2d9e21a024df5b9809574fd006d14 P 8ab11dc05044446fb867623234cd9a19: Neither blocks nor log segments found. Creating new log.
I20260812 07:08:50.868517 28336 tablet_bootstrap.cc:492] T d4c2d9e21a024df5b9809574fd006d14 P ad1b955a73c14e3fbf776cdb90e1f74a: No bootstrap required, opened a new log
I20260812 07:08:50.868788 28336 ts_tablet_manager.cc:1397] T d4c2d9e21a024df5b9809574fd006d14 P ad1b955a73c14e3fbf776cdb90e1f74a: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.006s
I20260812 07:08:50.869603 28338 log.cc:824] T d4c2d9e21a024df5b9809574fd006d14 P 8ab11dc05044446fb867623234cd9a19: Log is configured to *not* fsync() on all Append() calls
I20260812 07:08:50.872797 28338 tablet_bootstrap.cc:492] T d4c2d9e21a024df5b9809574fd006d14 P 8ab11dc05044446fb867623234cd9a19: No bootstrap required, opened a new log
I20260812 07:08:50.873095 28338 ts_tablet_manager.cc:1397] T d4c2d9e21a024df5b9809574fd006d14 P 8ab11dc05044446fb867623234cd9a19: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260812 07:08:50.877730 28335 raft_consensus.cc:348] T d4c2d9e21a024df5b9809574fd006d14 P cfa8a45cc47d4e97a3496e30c4d13366 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ab11dc05044446fb867623234cd9a19" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 43871 } } peers { permanent_uuid: "cfa8a45cc47d4e97a3496e30c4d13366" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 33025 } } peers { permanent_uuid: "ad1b955a73c14e3fbf776cdb90e1f74a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 36359 } }
I20260812 07:08:50.878289 28335 raft_consensus.cc:374] T d4c2d9e21a024df5b9809574fd006d14 P cfa8a45cc47d4e97a3496e30c4d13366 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:08:50.878396 28335 raft_consensus.cc:729] T d4c2d9e21a024df5b9809574fd006d14 P cfa8a45cc47d4e97a3496e30c4d13366 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfa8a45cc47d4e97a3496e30c4d13366, State: Initialized, Role: FOLLOWER
I20260812 07:08:50.879099 28335 consensus_queue.cc:260] T d4c2d9e21a024df5b9809574fd006d14 P cfa8a45cc47d4e97a3496e30c4d13366 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ab11dc05044446fb867623234cd9a19" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 43871 } } peers { permanent_uuid: "cfa8a45cc47d4e97a3496e30c4d13366" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 33025 } } peers { permanent_uuid: "ad1b955a73c14e3fbf776cdb90e1f74a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 36359 } }
I20260812 07:08:50.879652 28336 raft_consensus.cc:348] T d4c2d9e21a024df5b9809574fd006d14 P ad1b955a73c14e3fbf776cdb90e1f74a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ab11dc05044446fb867623234cd9a19" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 43871 } } peers { permanent_uuid: "cfa8a45cc47d4e97a3496e30c4d13366" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 33025 } } peers { permanent_uuid: "ad1b955a73c14e3fbf776cdb90e1f74a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 36359 } }
I20260812 07:08:50.880133 28336 raft_consensus.cc:374] T d4c2d9e21a024df5b9809574fd006d14 P ad1b955a73c14e3fbf776cdb90e1f74a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:08:50.880832 28338 raft_consensus.cc:348] T d4c2d9e21a024df5b9809574fd006d14 P 8ab11dc05044446fb867623234cd9a19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ab11dc05044446fb867623234cd9a19" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 43871 } } peers { permanent_uuid: "cfa8a45cc47d4e97a3496e30c4d13366" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 33025 } } peers { permanent_uuid: "ad1b955a73c14e3fbf776cdb90e1f74a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 36359 } }
I20260812 07:08:50.881126 28335 ts_tablet_manager.cc:1428] T d4c2d9e21a024df5b9809574fd006d14 P cfa8a45cc47d4e97a3496e30c4d13366: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260812 07:08:50.881192 28338 raft_consensus.cc:374] T d4c2d9e21a024df5b9809574fd006d14 P 8ab11dc05044446fb867623234cd9a19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:08:50.880259 28336 raft_consensus.cc:729] T d4c2d9e21a024df5b9809574fd006d14 P ad1b955a73c14e3fbf776cdb90e1f74a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad1b955a73c14e3fbf776cdb90e1f74a, State: Initialized, Role: FOLLOWER
I20260812 07:08:50.881312 28338 raft_consensus.cc:729] T d4c2d9e21a024df5b9809574fd006d14 P 8ab11dc05044446fb867623234cd9a19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ab11dc05044446fb867623234cd9a19, State: Initialized, Role: FOLLOWER
I20260812 07:08:50.881982 28336 consensus_queue.cc:260] T d4c2d9e21a024df5b9809574fd006d14 P ad1b955a73c14e3fbf776cdb90e1f74a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ab11dc05044446fb867623234cd9a19" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 43871 } } peers { permanent_uuid: "cfa8a45cc47d4e97a3496e30c4d13366" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 33025 } } peers { permanent_uuid: "ad1b955a73c14e3fbf776cdb90e1f74a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 36359 } }
I20260812 07:08:50.881982 28338 consensus_queue.cc:260] T d4c2d9e21a024df5b9809574fd006d14 P 8ab11dc05044446fb867623234cd9a19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ab11dc05044446fb867623234cd9a19" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 43871 } } peers { permanent_uuid: "cfa8a45cc47d4e97a3496e30c4d13366" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 33025 } } peers { permanent_uuid: "ad1b955a73c14e3fbf776cdb90e1f74a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 36359 } }
I20260812 07:08:50.886281 28338 ts_tablet_manager.cc:1428] T d4c2d9e21a024df5b9809574fd006d14 P 8ab11dc05044446fb867623234cd9a19: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260812 07:08:50.886852 28336 ts_tablet_manager.cc:1428] T d4c2d9e21a024df5b9809574fd006d14 P ad1b955a73c14e3fbf776cdb90e1f74a: Time spent starting tablet: real 0.018s user 0.013s sys 0.006s
I20260812 07:08:50.886907 28165 heartbeater.cc:503] Master 127.6.61.62:44631 was elected leader, sending a full tablet report...
I20260812 07:08:50.887331 28312 heartbeater.cc:503] Master 127.6.61.62:44631 was elected leader, sending a full tablet report...
W20260812 07:08:50.903045 28166 tablet.cc:2369] T d4c2d9e21a024df5b9809574fd006d14 P 8ab11dc05044446fb867623234cd9a19: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:08:50.960693 28342 raft_consensus.cc:482] T d4c2d9e21a024df5b9809574fd006d14 P ad1b955a73c14e3fbf776cdb90e1f74a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:08:50.961061 28342 raft_consensus.cc:504] T d4c2d9e21a024df5b9809574fd006d14 P ad1b955a73c14e3fbf776cdb90e1f74a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ab11dc05044446fb867623234cd9a19" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 43871 } } peers { permanent_uuid: "cfa8a45cc47d4e97a3496e30c4d13366" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 33025 } } peers { permanent_uuid: "ad1b955a73c14e3fbf776cdb90e1f74a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 36359 } }
I20260812 07:08:50.962984 28342 leader_election.cc:288] T d4c2d9e21a024df5b9809574fd006d14 P ad1b955a73c14e3fbf776cdb90e1f74a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ab11dc05044446fb867623234cd9a19 (127.6.61.2:43871), cfa8a45cc47d4e97a3496e30c4d13366 (127.6.61.1:33025)
I20260812 07:08:50.970060 28315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:50.963449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:33025 (local address 127.6.61.3:42447)
0812 07:08:50.963677 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:50.963701 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:50.963851 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:50.964388 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:50.964391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:50.964415 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:08:50.964627 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:50.964639 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:50.965991 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:50.965996 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:50.967661 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:50.967678 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:50.969257 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:50.969263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:50.969456 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:50.969481 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 07:08:50.969594 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 07:08:50.969735 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260812 07:08:50.970046 28332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:50.963582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:42447 (local address 127.6.61.1:33025)
0812 07:08:50.963792 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:50.963798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:50.963823 (+ 25us) server_negotiation.cc:407] Connection header received
0812 07:08:50.964075 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:50.964084 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:50.964229 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:50.964354 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:08:50.964784 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:50.965849 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:50.967844 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:50.969085 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:50.969234 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:50.969344 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:50.969622 (+ 278us) server_negotiation.cc:299] Negotiation successful
0812 07:08:50.969750 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 07:08:50.970860 28328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:50.963581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:38329 (local address 127.6.61.2:43871)
0812 07:08:50.963799 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:50.963805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:50.964583 (+ 778us) server_negotiation.cc:407] Connection header received
0812 07:08:50.964696 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:50.964704 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:50.964800 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:50.964871 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:08:50.965517 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:50.966599 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:50.968924 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:50.969906 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:50.970079 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:50.970186 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:50.970475 (+ 289us) server_negotiation.cc:299] Negotiation successful
0812 07:08:50.970609 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260812 07:08:50.971005 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:50.963448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:43871 (local address 127.6.61.3:38329)
0812 07:08:50.964442 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:50.964466 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:50.964592 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:50.964928 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:50.964932 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:50.964958 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:08:50.965305 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:50.965319 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:50.966808 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:50.966812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:50.968709 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:50.968727 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:50.970136 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:50.970143 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:50.970313 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:50.970337 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 07:08:50.970447 (+ 110us) client_negotiation.cc:240] Negotiation successful
0812 07:08:50.970574 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"mutex_wait_us":254,"thread_start_us":382,"threads_started":1}
I20260812 07:08:50.971571 27966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c2d9e21a024df5b9809574fd006d14" candidate_uuid: "ad1b955a73c14e3fbf776cdb90e1f74a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfa8a45cc47d4e97a3496e30c4d13366" is_pre_election: true
I20260812 07:08:50.971982 28118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c2d9e21a024df5b9809574fd006d14" candidate_uuid: "ad1b955a73c14e3fbf776cdb90e1f74a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ab11dc05044446fb867623234cd9a19" is_pre_election: true
I20260812 07:08:50.972142 27966 raft_consensus.cc:2454] T d4c2d9e21a024df5b9809574fd006d14 P cfa8a45cc47d4e97a3496e30c4d13366 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad1b955a73c14e3fbf776cdb90e1f74a in term 0.
I20260812 07:08:50.972470 28118 raft_consensus.cc:2454] T d4c2d9e21a024df5b9809574fd006d14 P 8ab11dc05044446fb867623234cd9a19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad1b955a73c14e3fbf776cdb90e1f74a in term 0.
I20260812 07:08:50.972954 28191 leader_election.cc:302] T d4c2d9e21a024df5b9809574fd006d14 P ad1b955a73c14e3fbf776cdb90e1f74a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad1b955a73c14e3fbf776cdb90e1f74a, cfa8a45cc47d4e97a3496e30c4d13366; no voters:
I20260812 07:08:50.973420 28342 raft_consensus.cc:2781] T d4c2d9e21a024df5b9809574fd006d14 P ad1b955a73c14e3fbf776cdb90e1f74a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:08:50.973587 28342 raft_consensus.cc:482] T d4c2d9e21a024df5b9809574fd006d14 P ad1b955a73c14e3fbf776cdb90e1f74a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:08:50.973696 28342 raft_consensus.cc:3037] T d4c2d9e21a024df5b9809574fd006d14 P ad1b955a73c14e3fbf776cdb90e1f74a [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:08:50.976011 28342 raft_consensus.cc:504] T d4c2d9e21a024df5b9809574fd006d14 P ad1b955a73c14e3fbf776cdb90e1f74a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ab11dc05044446fb867623234cd9a19" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 43871 } } peers { permanent_uuid: "cfa8a45cc47d4e97a3496e30c4d13366" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 33025 } } peers { permanent_uuid: "ad1b955a73c14e3fbf776cdb90e1f74a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 36359 } }
I20260812 07:08:50.976648 28342 leader_election.cc:288] T d4c2d9e21a024df5b9809574fd006d14 P ad1b955a73c14e3fbf776cdb90e1f74a [CANDIDATE]: Term 1 election: Requested vote from peers 8ab11dc05044446fb867623234cd9a19 (127.6.61.2:43871), cfa8a45cc47d4e97a3496e30c4d13366 (127.6.61.1:33025)
I20260812 07:08:50.977128 28118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c2d9e21a024df5b9809574fd006d14" candidate_uuid: "ad1b955a73c14e3fbf776cdb90e1f74a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ab11dc05044446fb867623234cd9a19"
I20260812 07:08:50.977221 27966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c2d9e21a024df5b9809574fd006d14" candidate_uuid: "ad1b955a73c14e3fbf776cdb90e1f74a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfa8a45cc47d4e97a3496e30c4d13366"
I20260812 07:08:50.977418 27966 raft_consensus.cc:3037] T d4c2d9e21a024df5b9809574fd006d14 P cfa8a45cc47d4e97a3496e30c4d13366 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:08:50.977418 28118 raft_consensus.cc:3037] T d4c2d9e21a024df5b9809574fd006d14 P 8ab11dc05044446fb867623234cd9a19 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:08:50.980803 28118 raft_consensus.cc:2454] T d4c2d9e21a024df5b9809574fd006d14 P 8ab11dc05044446fb867623234cd9a19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad1b955a73c14e3fbf776cdb90e1f74a in term 1.
I20260812 07:08:50.980803 27966 raft_consensus.cc:2454] T d4c2d9e21a024df5b9809574fd006d14 P cfa8a45cc47d4e97a3496e30c4d13366 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad1b955a73c14e3fbf776cdb90e1f74a in term 1.
I20260812 07:08:50.981300 28191 leader_election.cc:302] T d4c2d9e21a024df5b9809574fd006d14 P ad1b955a73c14e3fbf776cdb90e1f74a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad1b955a73c14e3fbf776cdb90e1f74a, cfa8a45cc47d4e97a3496e30c4d13366; no voters:
I20260812 07:08:50.981666 28342 raft_consensus.cc:2781] T d4c2d9e21a024df5b9809574fd006d14 P ad1b955a73c14e3fbf776cdb90e1f74a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:08:50.982187 28342 raft_consensus.cc:686] T d4c2d9e21a024df5b9809574fd006d14 P ad1b955a73c14e3fbf776cdb90e1f74a [term 1 LEADER]: Becoming Leader. State: Replica: ad1b955a73c14e3fbf776cdb90e1f74a, State: Running, Role: LEADER
I20260812 07:08:50.982764 28342 consensus_queue.cc:237] T d4c2d9e21a024df5b9809574fd006d14 P ad1b955a73c14e3fbf776cdb90e1f74a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ab11dc05044446fb867623234cd9a19" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 43871 } } peers { permanent_uuid: "cfa8a45cc47d4e97a3496e30c4d13366" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 33025 } } peers { permanent_uuid: "ad1b955a73c14e3fbf776cdb90e1f74a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 36359 } }
W20260812 07:08:50.988066 28313 tablet.cc:2369] T d4c2d9e21a024df5b9809574fd006d14 P ad1b955a73c14e3fbf776cdb90e1f74a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:08:50.987900 27805 catalog_manager.cc:5495] T d4c2d9e21a024df5b9809574fd006d14 P ad1b955a73c14e3fbf776cdb90e1f74a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad1b955a73c14e3fbf776cdb90e1f74a (127.6.61.3). New cstate: current_term: 1 leader_uuid: "ad1b955a73c14e3fbf776cdb90e1f74a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ab11dc05044446fb867623234cd9a19" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 43871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfa8a45cc47d4e97a3496e30c4d13366" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 33025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad1b955a73c14e3fbf776cdb90e1f74a" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 36359 } health_report { overall_health: HEALTHY } } }
I20260812 07:08:51.040381 28334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:51.034219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55576 (local address 127.6.61.3:36359)
0812 07:08:51.034440 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:51.034444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:51.034464 (+ 20us) server_negotiation.cc:407] Connection header received
0812 07:08:51.034703 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:51.034708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:51.034798 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:51.034932 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:08:51.035339 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:51.036119 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:51.037717 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:51.038267 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:51.038381 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:51.039018 (+ 637us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:08:51.039643 (+ 625us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:08:51.039792 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:51.040028 (+ 236us) server_negotiation.cc:299] Negotiation successful
0812 07:08:51.040087 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 07:08:51.050750 27966 raft_consensus.cc:1261] T d4c2d9e21a024df5b9809574fd006d14 P cfa8a45cc47d4e97a3496e30c4d13366 [term 1 FOLLOWER]: Refusing update from remote peer ad1b955a73c14e3fbf776cdb90e1f74a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:08:51.050750 28118 raft_consensus.cc:1261] T d4c2d9e21a024df5b9809574fd006d14 P 8ab11dc05044446fb867623234cd9a19 [term 1 FOLLOWER]: Refusing update from remote peer ad1b955a73c14e3fbf776cdb90e1f74a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:08:51.052064 28342 consensus_queue.cc:1035] T d4c2d9e21a024df5b9809574fd006d14 P ad1b955a73c14e3fbf776cdb90e1f74a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfa8a45cc47d4e97a3496e30c4d13366" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 33025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:08:51.052464 28345 consensus_queue.cc:1035] T d4c2d9e21a024df5b9809574fd006d14 P ad1b955a73c14e3fbf776cdb90e1f74a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ab11dc05044446fb867623234cd9a19" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 43871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 07:08:51.076642 28018 tablet.cc:2369] T d4c2d9e21a024df5b9809574fd006d14 P cfa8a45cc47d4e97a3496e30c4d13366: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:08:51.093237 27806 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:08:51.093614 27806 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:08:51.098337 27806 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82396e94dc00454c83436b77bd11c091: Sending DeleteTablet for 3 replicas of tablet d4c2d9e21a024df5b9809574fd006d14
I20260812 07:08:51.099874 27943 tablet_service.cc:1514] Processing DeleteTablet for tablet d4c2d9e21a024df5b9809574fd006d14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:08:51 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:52218
I20260812 07:08:51.099874 28095 tablet_service.cc:1514] Processing DeleteTablet for tablet d4c2d9e21a024df5b9809574fd006d14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:08:51 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:42446
I20260812 07:08:51.103088 28243 tablet_service.cc:1514] Processing DeleteTablet for tablet d4c2d9e21a024df5b9809574fd006d14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:08:51 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:55572
I20260812 07:08:51.106113 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 27877
I20260812 07:08:51.112648 28358 tablet_replica.cc:330] T d4c2d9e21a024df5b9809574fd006d14 P cfa8a45cc47d4e97a3496e30c4d13366: stopping tablet replica
I20260812 07:08:51.112886 28360 tablet_replica.cc:330] T d4c2d9e21a024df5b9809574fd006d14 P 8ab11dc05044446fb867623234cd9a19: stopping tablet replica
I20260812 07:08:51.113381 28358 raft_consensus.cc:2229] T d4c2d9e21a024df5b9809574fd006d14 P cfa8a45cc47d4e97a3496e30c4d13366 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:08:51.113381 28360 raft_consensus.cc:2229] T d4c2d9e21a024df5b9809574fd006d14 P 8ab11dc05044446fb867623234cd9a19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:08:51.113983 28358 raft_consensus.cc:2258] T d4c2d9e21a024df5b9809574fd006d14 P cfa8a45cc47d4e97a3496e30c4d13366 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:08:51.113984 28360 raft_consensus.cc:2258] T d4c2d9e21a024df5b9809574fd006d14 P 8ab11dc05044446fb867623234cd9a19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:08:51.116122 28358 ts_tablet_manager.cc:1905] T d4c2d9e21a024df5b9809574fd006d14 P cfa8a45cc47d4e97a3496e30c4d13366: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:08:51.121281 28358 ts_tablet_manager.cc:1918] T d4c2d9e21a024df5b9809574fd006d14 P cfa8a45cc47d4e97a3496e30c4d13366: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:08:51.121482 28358 log.cc:1196] T d4c2d9e21a024df5b9809574fd006d14 P cfa8a45cc47d4e97a3496e30c4d13366: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-0/wal/wals/d4c2d9e21a024df5b9809574fd006d14
I20260812 07:08:51.122202 28358 ts_tablet_manager.cc:1939] T d4c2d9e21a024df5b9809574fd006d14 P cfa8a45cc47d4e97a3496e30c4d13366: Deleting consensus metadata
I20260812 07:08:51.123730 28360 ts_tablet_manager.cc:1905] T d4c2d9e21a024df5b9809574fd006d14 P 8ab11dc05044446fb867623234cd9a19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:08:51.124743 27791 catalog_manager.cc:4841] TS cfa8a45cc47d4e97a3496e30c4d13366 (127.6.61.1:33025): tablet d4c2d9e21a024df5b9809574fd006d14 (table test-table [id=163d4aa3fb1742f89ed8c70a7a45ee98]) successfully deleted
I20260812 07:08:51.128151 28361 tablet_replica.cc:330] T d4c2d9e21a024df5b9809574fd006d14 P ad1b955a73c14e3fbf776cdb90e1f74a: stopping tablet replica
I20260812 07:08:51.128695 28361 raft_consensus.cc:2229] T d4c2d9e21a024df5b9809574fd006d14 P ad1b955a73c14e3fbf776cdb90e1f74a [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:08:51.129444 28361 raft_consensus.cc:2258] T d4c2d9e21a024df5b9809574fd006d14 P ad1b955a73c14e3fbf776cdb90e1f74a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:08:51.132071 28360 ts_tablet_manager.cc:1918] T d4c2d9e21a024df5b9809574fd006d14 P 8ab11dc05044446fb867623234cd9a19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:08:51.132320 28360 log.cc:1196] T d4c2d9e21a024df5b9809574fd006d14 P 8ab11dc05044446fb867623234cd9a19: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-1/wal/wals/d4c2d9e21a024df5b9809574fd006d14
I20260812 07:08:51.132982 28360 ts_tablet_manager.cc:1939] T d4c2d9e21a024df5b9809574fd006d14 P 8ab11dc05044446fb867623234cd9a19: Deleting consensus metadata
I20260812 07:08:51.135007 28361 ts_tablet_manager.cc:1905] T d4c2d9e21a024df5b9809574fd006d14 P ad1b955a73c14e3fbf776cdb90e1f74a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:08:51.135097 27792 catalog_manager.cc:4841] TS 8ab11dc05044446fb867623234cd9a19 (127.6.61.2:43871): tablet d4c2d9e21a024df5b9809574fd006d14 (table test-table [id=163d4aa3fb1742f89ed8c70a7a45ee98]) successfully deleted
Aug 12 07:08:51 dist-test-slave-69rg krb5kdc[27757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.1@KRBTEST.COM
I20260812 07:08:51.144444 28361 ts_tablet_manager.cc:1918] T d4c2d9e21a024df5b9809574fd006d14 P ad1b955a73c14e3fbf776cdb90e1f74a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:08:51.144886 28361 log.cc:1196] T d4c2d9e21a024df5b9809574fd006d14 P ad1b955a73c14e3fbf776cdb90e1f74a: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518208686283-6388-0/minicluster-data/ts-2/wal/wals/d4c2d9e21a024df5b9809574fd006d14
I20260812 07:08:51.145632 28361 ts_tablet_manager.cc:1939] T d4c2d9e21a024df5b9809574fd006d14 P ad1b955a73c14e3fbf776cdb90e1f74a: Deleting consensus metadata
I20260812 07:08:51.148064 27789 catalog_manager.cc:4841] TS ad1b955a73c14e3fbf776cdb90e1f74a (127.6.61.3:36359): tablet d4c2d9e21a024df5b9809574fd006d14 (table test-table [id=163d4aa3fb1742f89ed8c70a7a45ee98]) successfully deleted
I20260812 07:08:51.149082 28332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:51.110295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52226 (local address 127.6.61.1:33025)
0812 07:08:51.110557 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:51.110567 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:51.124521 (+ 13954us) server_negotiation.cc:407] Connection header received
0812 07:08:51.124685 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:51.124690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:51.124790 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:51.124877 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:51.129958 (+ 5081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:51.131119 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:51.133552 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:51.134291 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:51.134402 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:51.140373 (+ 5971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:51.140395 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:51.140416 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:08:51.140465 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:51.144994 (+ 4529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:51.146186 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:51.146197 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:51.146200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:51.146410 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:51.146988 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:51.146994 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:51.146997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:51.147520 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:51.147845 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:51.148399 (+ 554us) server_negotiation.cc:299] Negotiation successful
0812 07:08:51.148645 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260812 07:08:51.149885 28012 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:08:51.246984 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 27877
I20260812 07:08:51.270941 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 28024
Aug 12 07:08:51 dist-test-slave-69rg krb5kdc[27757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.2@KRBTEST.COM
I20260812 07:08:51.289390 28328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:51.271820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42452 (local address 127.6.61.2:43871)
0812 07:08:51.272046 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:51.272053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:51.272076 (+ 23us) server_negotiation.cc:407] Connection header received
0812 07:08:51.272186 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:51.272192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:51.272308 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:51.272428 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:51.273692 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:51.274893 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:51.276172 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:51.276809 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:51.276943 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:51.281572 (+ 4629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:51.281595 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:51.281619 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:08:51.281667 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:51.285609 (+ 3942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:51.286764 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:51.286775 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:51.286779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:51.286966 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:51.287532 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:51.287539 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:51.287542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:51.288088 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:51.288438 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:51.288719 (+ 281us) server_negotiation.cc:299] Negotiation successful
0812 07:08:51.288986 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 07:08:51.290227 28159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:08:51.382117 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 28024
I20260812 07:08:51.405493 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 28173
Aug 12 07:08:51 dist-test-slave-69rg krb5kdc[27757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.3@KRBTEST.COM
I20260812 07:08:51.423396 28334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:51.406651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55590 (local address 127.6.61.3:36359)
0812 07:08:51.406856 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:51.406865 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:51.406890 (+ 25us) server_negotiation.cc:407] Connection header received
0812 07:08:51.406961 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:51.406972 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:51.407037 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:51.407168 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:51.408796 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:51.409647 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:51.411177 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:51.411681 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:51.411842 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:51.416633 (+ 4791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:51.416648 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:51.416666 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:08:51.416705 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:51.420132 (+ 3427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:51.420895 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:51.420903 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:51.420905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:51.421082 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:51.421485 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:51.421490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:51.421493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:51.422096 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:51.422489 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:51.422777 (+ 288us) server_negotiation.cc:299] Negotiation successful
0812 07:08:51.423022 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 07:08:51.424198 28307 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:08:51.548267 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 28173
I20260812 07:08:51.573052 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 27772
I20260812 07:08:51.574407 27844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:08:51.687731 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 27772
2026-08-12T07:08:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4892 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-12T07:08:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:08:51Z Disabled control of system clock
I20260812 07:08:51.793105 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:45217
--webserver_interface=127.6.61.62
--webserver_port=0
--builtin_ntp_servers=127.6.61.20:41099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:45217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 07:08:52.166263 28370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:08:52.166664 28370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:08:52.166812 28370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:08:52.174530 28370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:08:52.174661 28370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:08:52.174750 28370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:08:52.174842 28370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:08:52.174918 28370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:08:52.184652 28370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:41099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:45217
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:45217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:08:52.186426 28370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:08:52.188635 28370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:08:52.201968 28375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:52.201918 28376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:52.201973 28379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:08:52.202934 28370 server_base.cc:1034] running on GCE node
I20260812 07:08:52.203774 28370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:08:52.205303 28370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:08:52.206566 28370 hybrid_clock.cc:648] HybridClock initialized: now 1786518532206515 us; error 34 us; skew 500 ppm
I20260812 07:08:52.207010 28370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 07:08:52.219085 28370 webserver.cc:460] Webserver started at http://127.6.61.62:42363/ using document root <none> and password file <none>
I20260812 07:08:52.219928 28370 fs_manager.cc:359] Metadata directory not provided
I20260812 07:08:52.220080 28370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:08:52.220438 28370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:08:52.223658 28370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "465d4913edbe45a497ddce45edbed5ef"
format_stamp: "Formatted at 2026-08-12 07:08:52 on dist-test-slave-69rg"
server_key: "61cc538cbada4b8055080b9430fe6b2f"
server_key_iv: "767f58644d840baa2c3961135dd563a2"
server_key_version: "encryptionkey@0"
I20260812 07:08:52.224540 28370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "465d4913edbe45a497ddce45edbed5ef"
format_stamp: "Formatted at 2026-08-12 07:08:52 on dist-test-slave-69rg"
server_key: "61cc538cbada4b8055080b9430fe6b2f"
server_key_iv: "767f58644d840baa2c3961135dd563a2"
server_key_version: "encryptionkey@0"
I20260812 07:08:52.230273 28370 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 07:08:52.234062 28385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:52.235440 28370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:08:52.235644 28370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "465d4913edbe45a497ddce45edbed5ef"
format_stamp: "Formatted at 2026-08-12 07:08:52 on dist-test-slave-69rg"
server_key: "61cc538cbada4b8055080b9430fe6b2f"
server_key_iv: "767f58644d840baa2c3961135dd563a2"
server_key_version: "encryptionkey@0"
I20260812 07:08:52.235874 28370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:08:52.254446 28370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:08:52.255424 28370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:08:52.255751 28370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:08:52.276763 28370 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:45217
I20260812 07:08:52.276809 28440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:45217 every 8 connection(s)
I20260812 07:08:52.278345 28370 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:08:52.282423 28441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:08:52.288798 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 28370
I20260812 07:08:52.289008 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:08:52.289507 6388 external_mini_cluster.cc:1428] Setting key 4be679a690f061aa7f2221be1ad44105
I20260812 07:08:52.295079 28441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 465d4913edbe45a497ddce45edbed5ef: Bootstrap starting.
I20260812 07:08:52.298635 28441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 465d4913edbe45a497ddce45edbed5ef: Neither blocks nor log segments found. Creating new log.
I20260812 07:08:52.299803 28441 log.cc:824] T 00000000000000000000000000000000 P 465d4913edbe45a497ddce45edbed5ef: Log is configured to *not* fsync() on all Append() calls
I20260812 07:08:52.303145 28441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 465d4913edbe45a497ddce45edbed5ef: No bootstrap required, opened a new log
I20260812 07:08:52.303689 28444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:52.291716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45108 (local address 127.6.61.62:45217)
0812 07:08:52.292882 (+ 1166us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:52.292894 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:52.292940 (+ 46us) server_negotiation.cc:407] Connection header received
0812 07:08:52.294803 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:52.294830 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:52.295352 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:52.295750 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:52.296339 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:52.297639 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:52.299154 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:52.299834 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:52.300035 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:52.301278 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:52.301286 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:52.301300 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:08:52.301902 (+ 602us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:52.301982 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:08:52.302004 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:52.302156 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:52.302379 (+ 223us) server_negotiation.cc:299] Negotiation successful
0812 07:08:52.302691 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":437,"threads_started":1}
I20260812 07:08:52.314297 28441 raft_consensus.cc:348] T 00000000000000000000000000000000 P 465d4913edbe45a497ddce45edbed5ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "465d4913edbe45a497ddce45edbed5ef" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 45217 } }
I20260812 07:08:52.314788 28441 raft_consensus.cc:374] T 00000000000000000000000000000000 P 465d4913edbe45a497ddce45edbed5ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:08:52.314909 28441 raft_consensus.cc:729] T 00000000000000000000000000000000 P 465d4913edbe45a497ddce45edbed5ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 465d4913edbe45a497ddce45edbed5ef, State: Initialized, Role: FOLLOWER
I20260812 07:08:52.315591 28441 consensus_queue.cc:260] T 00000000000000000000000000000000 P 465d4913edbe45a497ddce45edbed5ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "465d4913edbe45a497ddce45edbed5ef" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 45217 } }
I20260812 07:08:52.315883 28441 raft_consensus.cc:388] T 00000000000000000000000000000000 P 465d4913edbe45a497ddce45edbed5ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:08:52.316042 28441 raft_consensus.cc:482] T 00000000000000000000000000000000 P 465d4913edbe45a497ddce45edbed5ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:08:52.316243 28441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 465d4913edbe45a497ddce45edbed5ef [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:08:52.319011 28441 raft_consensus.cc:504] T 00000000000000000000000000000000 P 465d4913edbe45a497ddce45edbed5ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "465d4913edbe45a497ddce45edbed5ef" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 45217 } }
I20260812 07:08:52.319656 28441 leader_election.cc:302] T 00000000000000000000000000000000 P 465d4913edbe45a497ddce45edbed5ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 465d4913edbe45a497ddce45edbed5ef; no voters:
I20260812 07:08:52.320245 28441 leader_election.cc:288] T 00000000000000000000000000000000 P 465d4913edbe45a497ddce45edbed5ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:08:52.320425 28446 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 465d4913edbe45a497ddce45edbed5ef [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:08:52.321184 28446 raft_consensus.cc:686] T 00000000000000000000000000000000 P 465d4913edbe45a497ddce45edbed5ef [term 1 LEADER]: Becoming Leader. State: Replica: 465d4913edbe45a497ddce45edbed5ef, State: Running, Role: LEADER
I20260812 07:08:52.321750 28446 consensus_queue.cc:237] T 00000000000000000000000000000000 P 465d4913edbe45a497ddce45edbed5ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "465d4913edbe45a497ddce45edbed5ef" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 45217 } }
I20260812 07:08:52.322593 28441 sys_catalog.cc:564] T 00000000000000000000000000000000 P 465d4913edbe45a497ddce45edbed5ef [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:08:52.324831 28447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 465d4913edbe45a497ddce45edbed5ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "465d4913edbe45a497ddce45edbed5ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "465d4913edbe45a497ddce45edbed5ef" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 45217 } } }
I20260812 07:08:52.325285 28447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 465d4913edbe45a497ddce45edbed5ef [sys.catalog]: This master's current role is: LEADER
I20260812 07:08:52.324803 28448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 465d4913edbe45a497ddce45edbed5ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 465d4913edbe45a497ddce45edbed5ef. Latest consensus state: current_term: 1 leader_uuid: "465d4913edbe45a497ddce45edbed5ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "465d4913edbe45a497ddce45edbed5ef" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 45217 } } }
I20260812 07:08:52.325938 28448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 465d4913edbe45a497ddce45edbed5ef [sys.catalog]: This master's current role is: LEADER
I20260812 07:08:52.326988 28452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:08:52.332430 28452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:08:52.342458 28452 catalog_manager.cc:1347] Generated new cluster ID: f2b287adf82a4eeba5c441d2fbbc51b0
I20260812 07:08:52.342626 28452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:08:52.372213 28452 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:08:52.373658 28452 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:08:52.388036 28452 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 465d4913edbe45a497ddce45edbed5ef: Generated new TSK 0
I20260812 07:08:52.389081 28452 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:08:52.395483 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.1:0
--local_ip_for_outbound_sockets=127.6.61.1
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:45217
--builtin_ntp_servers=127.6.61.20:41099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 07:08:52.765748 28468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:08:52.766201 28468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:08:52.766359 28468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:08:52.773962 28468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:08:52.774109 28468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:08:52.774300 28468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.1
I20260812 07:08:52.783918 28468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:41099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:45217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.1
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:08:52.785318 28468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:08:52.787174 28468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:08:52.802328 28474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:52.802970 28475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:52.802991 28477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:08:52.804152 28468 server_base.cc:1034] running on GCE node
I20260812 07:08:52.804953 28468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:08:52.806311 28468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:08:52.807564 28468 hybrid_clock.cc:648] HybridClock initialized: now 1786518532807511 us; error 34 us; skew 500 ppm
I20260812 07:08:52.808012 28468 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 07:08:52.810851 28468 webserver.cc:460] Webserver started at http://127.6.61.1:40155/ using document root <none> and password file <none>
I20260812 07:08:52.811834 28468 fs_manager.cc:359] Metadata directory not provided
I20260812 07:08:52.811975 28468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:08:52.812381 28468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:08:52.815732 28468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-0/data/instance:
uuid: "2ffe0a592e28441a988641a153cf4a79"
format_stamp: "Formatted at 2026-08-12 07:08:52 on dist-test-slave-69rg"
server_key: "054940d78f8fe7d68f6efabf0b0eee26"
server_key_iv: "115973d114d335893d3c1308c9502c83"
server_key_version: "encryptionkey@0"
I20260812 07:08:52.816756 28468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance:
uuid: "2ffe0a592e28441a988641a153cf4a79"
format_stamp: "Formatted at 2026-08-12 07:08:52 on dist-test-slave-69rg"
server_key: "054940d78f8fe7d68f6efabf0b0eee26"
server_key_iv: "115973d114d335893d3c1308c9502c83"
server_key_version: "encryptionkey@0"
I20260812 07:08:52.822614 28468 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 07:08:52.826484 28483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:52.827972 28468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260812 07:08:52.828153 28468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-0/wal
uuid: "2ffe0a592e28441a988641a153cf4a79"
format_stamp: "Formatted at 2026-08-12 07:08:52 on dist-test-slave-69rg"
server_key: "054940d78f8fe7d68f6efabf0b0eee26"
server_key_iv: "115973d114d335893d3c1308c9502c83"
server_key_version: "encryptionkey@0"
I20260812 07:08:52.828377 28468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:08:52.850514 28468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:08:52.851527 28468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:08:52.851893 28468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:08:52.853062 28468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:08:52.854981 28468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:08:52.855119 28468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:52.855268 28468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:08:52.855382 28468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:52.894289 28468 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.1:40355
I20260812 07:08:52.894400 28599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.1:40355 every 8 connection(s)
I20260812 07:08:52.895835 28468 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
I20260812 07:08:52.902524 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 28468
I20260812 07:08:52.902740 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance
I20260812 07:08:52.903272 6388 external_mini_cluster.cc:1428] Setting key 2f636afda5a5cdfca544d0952124c40c
I20260812 07:08:52.906294 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.2:0
--local_ip_for_outbound_sockets=127.6.61.2
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:45217
--builtin_ntp_servers=127.6.61.20:41099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 07:08:52.909894 28605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:52.898877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:48401 (local address 127.6.61.62:45217)
0812 07:08:52.899373 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:52.899378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:52.900475 (+ 1097us) server_negotiation.cc:407] Connection header received
0812 07:08:52.902552 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:52.902557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:52.902659 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:52.902816 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:52.904044 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:52.905063 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:52.906905 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:52.907315 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:52.907459 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:52.908952 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:52.908957 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:52.908960 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:08:52.908978 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:52.909021 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:08:52.909033 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:52.909141 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:52.909391 (+ 250us) server_negotiation.cc:299] Negotiation successful
0812 07:08:52.909498 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":258,"threads_started":1}
I20260812 07:08:52.910962 28606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:52.899365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:45217 (local address 127.6.61.1:48401)
0812 07:08:52.900286 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:52.900338 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:52.902209 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:52.903070 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:52.903078 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:52.903177 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:08:52.903851 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:52.903871 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:52.905238 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:52.905242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:52.906748 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:52.906763 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:52.907558 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:52.907565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:52.907811 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:52.908576 (+ 765us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:08:52.908579 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:08:52.908625 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:08:52.908724 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:08:52.909134 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:08:52.909147 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:08:52.909157 (+ 10us) client_negotiation.cc:769] Sending connection context
0812 07:08:52.909364 (+ 207us) client_negotiation.cc:240] Negotiation successful
0812 07:08:52.909867 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":310,"threads_started":1}
I20260812 07:08:52.912794 28600 heartbeater.cc:347] Connected to a master server at 127.6.61.62:45217
I20260812 07:08:52.913147 28600 heartbeater.cc:464] Registering TS with master...
I20260812 07:08:52.914073 28600 heartbeater.cc:511] Master 127.6.61.62:45217 requested a full tablet report, sending...
I20260812 07:08:52.916389 28405 ts_manager.cc:194] Registered new tserver with Master: 2ffe0a592e28441a988641a153cf4a79 (127.6.61.1:40355)
I20260812 07:08:52.918022 28405 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.61.1:48401
W20260812 07:08:53.264621 28607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:08:53.264930 28607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:08:53.265077 28607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:08:53.272523 28607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:08:53.272639 28607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:08:53.272814 28607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.2
I20260812 07:08:53.282212 28607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:41099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:45217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.2
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:08:53.283528 28607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:08:53.285322 28607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:08:53.299381 28612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:53.299433 28613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:53.299381 28615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:08:53.300515 28607 server_base.cc:1034] running on GCE node
I20260812 07:08:53.301283 28607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:08:53.302517 28607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:08:53.303807 28607 hybrid_clock.cc:648] HybridClock initialized: now 1786518533303731 us; error 62 us; skew 500 ppm
I20260812 07:08:53.304189 28607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 07:08:53.307045 28607 webserver.cc:460] Webserver started at http://127.6.61.2:38273/ using document root <none> and password file <none>
I20260812 07:08:53.307813 28607 fs_manager.cc:359] Metadata directory not provided
I20260812 07:08:53.307969 28607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:08:53.308338 28607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:08:53.311545 28607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-1/data/instance:
uuid: "cdb1bb1bac604d2ca2b4a840508b472c"
format_stamp: "Formatted at 2026-08-12 07:08:53 on dist-test-slave-69rg"
server_key: "857180ee5536c1b678324d0a3003a149"
server_key_iv: "d906f134081a7c58bf810d7dc2cc8478"
server_key_version: "encryptionkey@0"
I20260812 07:08:53.312379 28607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance:
uuid: "cdb1bb1bac604d2ca2b4a840508b472c"
format_stamp: "Formatted at 2026-08-12 07:08:53 on dist-test-slave-69rg"
server_key: "857180ee5536c1b678324d0a3003a149"
server_key_iv: "d906f134081a7c58bf810d7dc2cc8478"
server_key_version: "encryptionkey@0"
I20260812 07:08:53.317907 28607 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 07:08:53.321740 28621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:53.323263 28607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:08:53.323429 28607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-1/wal
uuid: "cdb1bb1bac604d2ca2b4a840508b472c"
format_stamp: "Formatted at 2026-08-12 07:08:53 on dist-test-slave-69rg"
server_key: "857180ee5536c1b678324d0a3003a149"
server_key_iv: "d906f134081a7c58bf810d7dc2cc8478"
server_key_version: "encryptionkey@0"
I20260812 07:08:53.323657 28607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:08:53.344918 28607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:08:53.345791 28607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:08:53.346171 28607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:08:53.347195 28607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:08:53.348793 28607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:08:53.348932 28607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:53.349052 28607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:08:53.349138 28607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:53.384086 28607 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.2:34259
I20260812 07:08:53.384182 28739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.2:34259 every 8 connection(s)
I20260812 07:08:53.385424 28607 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
I20260812 07:08:53.392568 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 28607
I20260812 07:08:53.392807 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance
I20260812 07:08:53.393265 6388 external_mini_cluster.cc:1428] Setting key af5baac47f1ceb9c521867201a298b63
I20260812 07:08:53.396281 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.3:0
--local_ip_for_outbound_sockets=127.6.61.3
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:45217
--builtin_ntp_servers=127.6.61.20:41099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 07:08:53.400403 28605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:53.388445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:52347 (local address 127.6.61.62:45217)
0812 07:08:53.388651 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:53.388656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:53.390033 (+ 1377us) server_negotiation.cc:407] Connection header received
0812 07:08:53.392068 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:53.392075 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:53.392208 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:53.392475 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:53.393765 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:53.394762 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:53.396627 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:53.397368 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:53.397531 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:53.399562 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:53.399568 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:53.399571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:08:53.399592 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:53.399632 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:08:53.399645 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:53.399737 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:53.400040 (+ 303us) server_negotiation.cc:299] Negotiation successful
0812 07:08:53.400129 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 07:08:53.401973 28744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:53.388868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:45217 (local address 127.6.61.2:52347)
0812 07:08:53.389797 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:53.389913 (+ 116us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:53.391759 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:53.392757 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:53.392770 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:53.392901 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:08:53.393577 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:53.393600 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:53.394938 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:53.394943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:53.396446 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:53.396470 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:53.397877 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:53.397889 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:53.398293 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:53.399135 (+ 842us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:08:53.399141 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:08:53.399201 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:08:53.399336 (+ 135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:08:53.399763 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:08:53.399776 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:08:53.399787 (+ 11us) client_negotiation.cc:769] Sending connection context
0812 07:08:53.399990 (+ 203us) client_negotiation.cc:240] Negotiation successful
0812 07:08:53.400474 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":284,"threads_started":1}
I20260812 07:08:53.403481 28742 heartbeater.cc:347] Connected to a master server at 127.6.61.62:45217
I20260812 07:08:53.403836 28742 heartbeater.cc:464] Registering TS with master...
I20260812 07:08:53.404857 28742 heartbeater.cc:511] Master 127.6.61.62:45217 requested a full tablet report, sending...
I20260812 07:08:53.406714 28405 ts_manager.cc:194] Registered new tserver with Master: cdb1bb1bac604d2ca2b4a840508b472c (127.6.61.2:34259)
I20260812 07:08:53.407647 28405 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.61.2:52347
W20260812 07:08:53.758273 28745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:08:53.758635 28745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:08:53.758809 28745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:08:53.766420 28745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:08:53.766589 28745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:08:53.766777 28745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.3
I20260812 07:08:53.776126 28745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:41099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:45217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.3
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:08:53.777527 28745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:08:53.779441 28745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:08:53.792593 28750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:53.792631 28753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:53.792675 28751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:08:53.794277 28745 server_base.cc:1034] running on GCE node
I20260812 07:08:53.795107 28745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:08:53.796430 28745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:08:53.797878 28745 hybrid_clock.cc:648] HybridClock initialized: now 1786518533797778 us; error 62 us; skew 500 ppm
I20260812 07:08:53.798297 28745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 07:08:53.801025 28745 webserver.cc:460] Webserver started at http://127.6.61.3:32919/ using document root <none> and password file <none>
I20260812 07:08:53.801789 28745 fs_manager.cc:359] Metadata directory not provided
I20260812 07:08:53.801990 28745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:08:53.802347 28745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:08:53.805471 28745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-2/data/instance:
uuid: "4fdf45109f154799955cae89afaca8fe"
format_stamp: "Formatted at 2026-08-12 07:08:53 on dist-test-slave-69rg"
server_key: "419f7856d95a09f58a5a3486b6478ec0"
server_key_iv: "486c7c85c78cd628d99da256170e5d52"
server_key_version: "encryptionkey@0"
I20260812 07:08:53.806402 28745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance:
uuid: "4fdf45109f154799955cae89afaca8fe"
format_stamp: "Formatted at 2026-08-12 07:08:53 on dist-test-slave-69rg"
server_key: "419f7856d95a09f58a5a3486b6478ec0"
server_key_iv: "486c7c85c78cd628d99da256170e5d52"
server_key_version: "encryptionkey@0"
I20260812 07:08:53.811908 28745 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 07:08:53.815937 28759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:53.817464 28745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 07:08:53.817680 28745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-2/wal
uuid: "4fdf45109f154799955cae89afaca8fe"
format_stamp: "Formatted at 2026-08-12 07:08:53 on dist-test-slave-69rg"
server_key: "419f7856d95a09f58a5a3486b6478ec0"
server_key_iv: "486c7c85c78cd628d99da256170e5d52"
server_key_version: "encryptionkey@0"
I20260812 07:08:53.818001 28745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:08:53.863009 28745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:08:53.864025 28745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:08:53.864362 28745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:08:53.865475 28745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:08:53.867214 28745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:08:53.867354 28745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:53.867511 28745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:08:53.867614 28745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:53.906560 28745 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.3:44939
I20260812 07:08:53.906664 28882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.3:44939 every 8 connection(s)
I20260812 07:08:53.908131 28745 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
I20260812 07:08:53.913292 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 28745
I20260812 07:08:53.913497 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance
I20260812 07:08:53.914054 6388 external_mini_cluster.cc:1428] Setting key 6bb5527cf37023dfa0701eac9c6da4ea
I20260812 07:08:53.921370 28600 heartbeater.cc:503] Master 127.6.61.62:45217 was elected leader, sending a full tablet report...
I20260812 07:08:53.922924 28885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:53.911231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:37661 (local address 127.6.61.62:45217)
0812 07:08:53.911749 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:53.911755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:53.912925 (+ 1170us) server_negotiation.cc:407] Connection header received
0812 07:08:53.915294 (+ 2369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:53.915301 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:53.915406 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:53.915535 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:53.916709 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:53.917563 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:53.919429 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:53.919884 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:53.920031 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:53.921789 (+ 1758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:53.921796 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:53.921801 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:08:53.921827 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:53.921901 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:08:53.921922 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:53.922025 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:53.922333 (+ 308us) server_negotiation.cc:299] Negotiation successful
0812 07:08:53.922438 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":254,"threads_started":1}
I20260812 07:08:53.924091 28886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:53.911806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:45217 (local address 127.6.61.3:37661)
0812 07:08:53.912737 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:53.912782 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:53.914987 (+ 2205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:53.915805 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:53.915814 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:53.915913 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:08:53.916520 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:53.916538 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:53.917764 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:53.917769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:53.919242 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:53.919257 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:53.920152 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:53.920161 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:53.920425 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:53.921299 (+ 874us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:08:53.921303 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:08:53.921367 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:08:53.921489 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:08:53.922042 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:08:53.922055 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:08:53.922063 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 07:08:53.922288 (+ 225us) client_negotiation.cc:240] Negotiation successful
0812 07:08:53.922774 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":276,"threads_started":1}
I20260812 07:08:53.925662 28883 heartbeater.cc:347] Connected to a master server at 127.6.61.62:45217
I20260812 07:08:53.926040 28883 heartbeater.cc:464] Registering TS with master...
I20260812 07:08:53.926865 28883 heartbeater.cc:511] Master 127.6.61.62:45217 requested a full tablet report, sending...
I20260812 07:08:53.928674 28405 ts_manager.cc:194] Registered new tserver with Master: 4fdf45109f154799955cae89afaca8fe (127.6.61.3:44939)
I20260812 07:08:53.929738 6388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 07:08:53.929764 28405 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.61.3:37661
I20260812 07:08:53.942066 28885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:53.935641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45114 (local address 127.6.61.62:45217)
0812 07:08:53.935826 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:53.935831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:53.936356 (+ 525us) server_negotiation.cc:407] Connection header received
0812 07:08:53.936505 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:53.936509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:53.936577 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:53.936721 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:53.937170 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:53.937997 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:53.939576 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:53.940028 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:53.940131 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:53.941349 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:53.941354 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:53.941357 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:08:53.941375 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:53.941407 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:08:53.941418 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:53.941531 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:53.941704 (+ 173us) server_negotiation.cc:299] Negotiation successful
0812 07:08:53.941752 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 07:08:53.946486 28405 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:08:53.954216 28405 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:08:53.989728 28900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:53.968213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55938 (local address 127.6.61.3:44939)
0812 07:08:53.980444 (+ 12231us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:53.980455 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:53.980497 (+ 42us) server_negotiation.cc:407] Connection header received
0812 07:08:53.980595 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:53.980606 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:53.980841 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:53.981050 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:08:53.981859 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:53.983870 (+ 2011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:53.986309 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:53.987911 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:53.988048 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:53.988373 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:53.989154 (+ 781us) server_negotiation.cc:299] Negotiation successful
0812 07:08:53.989330 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12029,"thread_start_us":11840,"threads_started":1}
I20260812 07:08:53.989763 28898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:53.967548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53364 (local address 127.6.61.2:34259)
0812 07:08:53.979765 (+ 12217us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:53.979778 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:53.979820 (+ 42us) server_negotiation.cc:407] Connection header received
0812 07:08:53.979957 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:53.979967 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:53.980285 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:53.980525 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:08:53.981362 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:53.982945 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:53.986485 (+ 3540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:53.988161 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:53.988318 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:53.988560 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:53.989216 (+ 656us) server_negotiation.cc:299] Negotiation successful
0812 07:08:53.989395 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12061,"thread_start_us":11965,"threads_started":1}
I20260812 07:08:53.990108 28902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:53.975722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59382 (local address 127.6.61.1:40355)
0812 07:08:53.980557 (+ 4835us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:53.980565 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:53.980593 (+ 28us) server_negotiation.cc:407] Connection header received
0812 07:08:53.980697 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:53.980707 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:53.980953 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:53.981186 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:08:53.982113 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:53.983900 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:53.986965 (+ 3065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:53.988594 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:53.988774 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:53.989064 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:53.989525 (+ 461us) server_negotiation.cc:299] Negotiation successful
0812 07:08:53.989697 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4664,"thread_start_us":4543,"threads_started":1}
I20260812 07:08:53.989794 28897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:53.975340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:40355 (local address 127.0.0.1:59382)
0812 07:08:53.979021 (+ 3681us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:53.979045 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:53.979203 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:53.981394 (+ 2191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:53.981400 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:53.981431 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:08:53.981929 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:53.981940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:53.984112 (+ 2172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:53.984118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:53.986725 (+ 2607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:53.986740 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:53.988857 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:53.988866 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:53.989075 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:53.989128 (+ 53us) client_negotiation.cc:769] Sending connection context
0812 07:08:53.989217 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 07:08:53.989364 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3493,"mutex_wait_us":1,"thread_start_us":2930,"threads_started":1}
I20260812 07:08:53.990823 28899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:53.967514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:44939 (local address 127.0.0.1:55938)
0812 07:08:53.974633 (+ 7119us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:53.974704 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:53.974968 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:53.981051 (+ 6083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:53.981056 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:53.981085 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:08:53.981606 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:53.981619 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:53.984018 (+ 2399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:53.984026 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:53.985991 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:53.986006 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:53.988511 (+ 2505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:53.988517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:53.988755 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:53.988882 (+ 127us) client_negotiation.cc:769] Sending connection context
0812 07:08:53.989024 (+ 142us) client_negotiation.cc:240] Negotiation successful
0812 07:08:53.990190 (+ 1166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6717,"mutex_wait_us":203,"thread_start_us":11145,"threads_started":1}
I20260812 07:08:53.991026 28901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:53.967562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:34259 (local address 127.0.0.1:53364)
0812 07:08:53.977889 (+ 10327us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:53.977927 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:53.978140 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:53.980571 (+ 2431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:53.980580 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:53.980615 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:08:53.981084 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:53.981102 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:53.983104 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:53.983115 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:53.986222 (+ 3107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:53.986240 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:53.988655 (+ 2415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:53.988661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:53.988911 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:53.988940 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 07:08:53.989024 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 07:08:53.989237 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10099,"mutex_wait_us":154,"thread_start_us":6485,"threads_started":1}
I20260812 07:08:53.994496 28532 tablet_service.cc:1467] Processing CreateTablet for tablet b9d8fee246e74aa08e4618615ac5c013 (DEFAULT_TABLE table=test-table [id=0a29b50de1554043ba652182720627d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:08:53.994510 28670 tablet_service.cc:1467] Processing CreateTablet for tablet b9d8fee246e74aa08e4618615ac5c013 (DEFAULT_TABLE table=test-table [id=0a29b50de1554043ba652182720627d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:08:53.994486 28811 tablet_service.cc:1467] Processing CreateTablet for tablet b9d8fee246e74aa08e4618615ac5c013 (DEFAULT_TABLE table=test-table [id=0a29b50de1554043ba652182720627d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:08:53.995926 28811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d8fee246e74aa08e4618615ac5c013. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:08:53.995978 28532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d8fee246e74aa08e4618615ac5c013. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:08:53.995975 28670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d8fee246e74aa08e4618615ac5c013. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:08:54.009470 28903 tablet_bootstrap.cc:492] T b9d8fee246e74aa08e4618615ac5c013 P 4fdf45109f154799955cae89afaca8fe: Bootstrap starting.
I20260812 07:08:54.012861 28904 tablet_bootstrap.cc:492] T b9d8fee246e74aa08e4618615ac5c013 P 2ffe0a592e28441a988641a153cf4a79: Bootstrap starting.
I20260812 07:08:54.012871 28905 tablet_bootstrap.cc:492] T b9d8fee246e74aa08e4618615ac5c013 P cdb1bb1bac604d2ca2b4a840508b472c: Bootstrap starting.
I20260812 07:08:54.014703 28903 tablet_bootstrap.cc:654] T b9d8fee246e74aa08e4618615ac5c013 P 4fdf45109f154799955cae89afaca8fe: Neither blocks nor log segments found. Creating new log.
I20260812 07:08:54.016371 28904 tablet_bootstrap.cc:654] T b9d8fee246e74aa08e4618615ac5c013 P 2ffe0a592e28441a988641a153cf4a79: Neither blocks nor log segments found. Creating new log.
I20260812 07:08:54.016530 28903 log.cc:824] T b9d8fee246e74aa08e4618615ac5c013 P 4fdf45109f154799955cae89afaca8fe: Log is configured to *not* fsync() on all Append() calls
I20260812 07:08:54.017428 28905 tablet_bootstrap.cc:654] T b9d8fee246e74aa08e4618615ac5c013 P cdb1bb1bac604d2ca2b4a840508b472c: Neither blocks nor log segments found. Creating new log.
I20260812 07:08:54.017589 28904 log.cc:824] T b9d8fee246e74aa08e4618615ac5c013 P 2ffe0a592e28441a988641a153cf4a79: Log is configured to *not* fsync() on all Append() calls
I20260812 07:08:54.018750 28905 log.cc:824] T b9d8fee246e74aa08e4618615ac5c013 P cdb1bb1bac604d2ca2b4a840508b472c: Log is configured to *not* fsync() on all Append() calls
I20260812 07:08:54.020200 28903 tablet_bootstrap.cc:492] T b9d8fee246e74aa08e4618615ac5c013 P 4fdf45109f154799955cae89afaca8fe: No bootstrap required, opened a new log
I20260812 07:08:54.020550 28903 ts_tablet_manager.cc:1397] T b9d8fee246e74aa08e4618615ac5c013 P 4fdf45109f154799955cae89afaca8fe: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260812 07:08:54.022328 28905 tablet_bootstrap.cc:492] T b9d8fee246e74aa08e4618615ac5c013 P cdb1bb1bac604d2ca2b4a840508b472c: No bootstrap required, opened a new log
I20260812 07:08:54.022718 28905 ts_tablet_manager.cc:1397] T b9d8fee246e74aa08e4618615ac5c013 P cdb1bb1bac604d2ca2b4a840508b472c: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260812 07:08:54.033303 28904 tablet_bootstrap.cc:492] T b9d8fee246e74aa08e4618615ac5c013 P 2ffe0a592e28441a988641a153cf4a79: No bootstrap required, opened a new log
I20260812 07:08:54.033152 28903 raft_consensus.cc:348] T b9d8fee246e74aa08e4618615ac5c013 P 4fdf45109f154799955cae89afaca8fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb1bb1bac604d2ca2b4a840508b472c" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 34259 } } peers { permanent_uuid: "4fdf45109f154799955cae89afaca8fe" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 44939 } } peers { permanent_uuid: "2ffe0a592e28441a988641a153cf4a79" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 40355 } }
I20260812 07:08:54.033669 28904 ts_tablet_manager.cc:1397] T b9d8fee246e74aa08e4618615ac5c013 P 2ffe0a592e28441a988641a153cf4a79: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.008s
I20260812 07:08:54.033721 28903 raft_consensus.cc:374] T b9d8fee246e74aa08e4618615ac5c013 P 4fdf45109f154799955cae89afaca8fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:08:54.033890 28903 raft_consensus.cc:729] T b9d8fee246e74aa08e4618615ac5c013 P 4fdf45109f154799955cae89afaca8fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fdf45109f154799955cae89afaca8fe, State: Initialized, Role: FOLLOWER
I20260812 07:08:54.034337 28905 raft_consensus.cc:348] T b9d8fee246e74aa08e4618615ac5c013 P cdb1bb1bac604d2ca2b4a840508b472c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb1bb1bac604d2ca2b4a840508b472c" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 34259 } } peers { permanent_uuid: "4fdf45109f154799955cae89afaca8fe" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 44939 } } peers { permanent_uuid: "2ffe0a592e28441a988641a153cf4a79" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 40355 } }
I20260812 07:08:54.034895 28905 raft_consensus.cc:374] T b9d8fee246e74aa08e4618615ac5c013 P cdb1bb1bac604d2ca2b4a840508b472c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:08:54.035008 28905 raft_consensus.cc:729] T b9d8fee246e74aa08e4618615ac5c013 P cdb1bb1bac604d2ca2b4a840508b472c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdb1bb1bac604d2ca2b4a840508b472c, State: Initialized, Role: FOLLOWER
I20260812 07:08:54.035324 28903 consensus_queue.cc:260] T b9d8fee246e74aa08e4618615ac5c013 P 4fdf45109f154799955cae89afaca8fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb1bb1bac604d2ca2b4a840508b472c" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 34259 } } peers { permanent_uuid: "4fdf45109f154799955cae89afaca8fe" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 44939 } } peers { permanent_uuid: "2ffe0a592e28441a988641a153cf4a79" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 40355 } }
I20260812 07:08:54.035745 28905 consensus_queue.cc:260] T b9d8fee246e74aa08e4618615ac5c013 P cdb1bb1bac604d2ca2b4a840508b472c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb1bb1bac604d2ca2b4a840508b472c" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 34259 } } peers { permanent_uuid: "4fdf45109f154799955cae89afaca8fe" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 44939 } } peers { permanent_uuid: "2ffe0a592e28441a988641a153cf4a79" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 40355 } }
I20260812 07:08:54.036723 28883 heartbeater.cc:503] Master 127.6.61.62:45217 was elected leader, sending a full tablet report...
I20260812 07:08:54.037726 28903 ts_tablet_manager.cc:1428] T b9d8fee246e74aa08e4618615ac5c013 P 4fdf45109f154799955cae89afaca8fe: Time spent starting tablet: real 0.017s user 0.015s sys 0.001s
I20260812 07:08:54.039510 28905 ts_tablet_manager.cc:1428] T b9d8fee246e74aa08e4618615ac5c013 P cdb1bb1bac604d2ca2b4a840508b472c: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260812 07:08:54.040019 28742 heartbeater.cc:503] Master 127.6.61.62:45217 was elected leader, sending a full tablet report...
I20260812 07:08:54.045488 28904 raft_consensus.cc:348] T b9d8fee246e74aa08e4618615ac5c013 P 2ffe0a592e28441a988641a153cf4a79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb1bb1bac604d2ca2b4a840508b472c" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 34259 } } peers { permanent_uuid: "4fdf45109f154799955cae89afaca8fe" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 44939 } } peers { permanent_uuid: "2ffe0a592e28441a988641a153cf4a79" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 40355 } }
I20260812 07:08:54.045954 28904 raft_consensus.cc:374] T b9d8fee246e74aa08e4618615ac5c013 P 2ffe0a592e28441a988641a153cf4a79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:08:54.046041 28904 raft_consensus.cc:729] T b9d8fee246e74aa08e4618615ac5c013 P 2ffe0a592e28441a988641a153cf4a79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ffe0a592e28441a988641a153cf4a79, State: Initialized, Role: FOLLOWER
I20260812 07:08:54.046677 28904 consensus_queue.cc:260] T b9d8fee246e74aa08e4618615ac5c013 P 2ffe0a592e28441a988641a153cf4a79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb1bb1bac604d2ca2b4a840508b472c" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 34259 } } peers { permanent_uuid: "4fdf45109f154799955cae89afaca8fe" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 44939 } } peers { permanent_uuid: "2ffe0a592e28441a988641a153cf4a79" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 40355 } }
I20260812 07:08:54.048792 28904 ts_tablet_manager.cc:1428] T b9d8fee246e74aa08e4618615ac5c013 P 2ffe0a592e28441a988641a153cf4a79: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
W20260812 07:08:54.137882 28743 tablet.cc:2369] T b9d8fee246e74aa08e4618615ac5c013 P cdb1bb1bac604d2ca2b4a840508b472c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 07:08:54.148622 28604 tablet.cc:2369] T b9d8fee246e74aa08e4618615ac5c013 P 2ffe0a592e28441a988641a153cf4a79: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 07:08:54.160478 28884 tablet.cc:2369] T b9d8fee246e74aa08e4618615ac5c013 P 4fdf45109f154799955cae89afaca8fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:08:54.356693 28909 raft_consensus.cc:482] T b9d8fee246e74aa08e4618615ac5c013 P 4fdf45109f154799955cae89afaca8fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:08:54.357043 28909 raft_consensus.cc:504] T b9d8fee246e74aa08e4618615ac5c013 P 4fdf45109f154799955cae89afaca8fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb1bb1bac604d2ca2b4a840508b472c" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 34259 } } peers { permanent_uuid: "4fdf45109f154799955cae89afaca8fe" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 44939 } } peers { permanent_uuid: "2ffe0a592e28441a988641a153cf4a79" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 40355 } }
I20260812 07:08:54.358896 28909 leader_election.cc:288] T b9d8fee246e74aa08e4618615ac5c013 P 4fdf45109f154799955cae89afaca8fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdb1bb1bac604d2ca2b4a840508b472c (127.6.61.2:34259), 2ffe0a592e28441a988641a153cf4a79 (127.6.61.1:40355)
I20260812 07:08:54.365525 28898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:54.359415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:58239 (local address 127.6.61.2:34259)
0812 07:08:54.359633 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:54.359641 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:54.359669 (+ 28us) server_negotiation.cc:407] Connection header received
0812 07:08:54.359784 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:54.359792 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:54.359906 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:54.360017 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:08:54.360588 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:54.361595 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:54.363555 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:54.364598 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:54.364700 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:54.364772 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:54.365165 (+ 393us) server_negotiation.cc:299] Negotiation successful
0812 07:08:54.365265 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 07:08:54.365619 28886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:54.359136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:34259 (local address 127.6.61.3:58239)
0812 07:08:54.359331 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:54.359352 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:54.359492 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:54.360142 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:54.360148 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:54.360181 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:08:54.360412 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:54.360424 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:54.361730 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:54.361736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:54.363361 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:54.363375 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:54.364759 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:54.364766 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:54.364988 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:54.365027 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 07:08:54.365131 (+ 104us) client_negotiation.cc:240] Negotiation successful
0812 07:08:54.365286 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":4}
I20260812 07:08:54.366340 28902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:54.359967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:54491 (local address 127.6.61.1:40355)
0812 07:08:54.360154 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:54.360160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:54.360322 (+ 162us) server_negotiation.cc:407] Connection header received
0812 07:08:54.360445 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:54.360451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:54.360549 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:54.360638 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:08:54.361264 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:54.362227 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:54.364259 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:54.365343 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:54.365472 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:54.365565 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:54.365981 (+ 416us) server_negotiation.cc:299] Negotiation successful
0812 07:08:54.366118 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 07:08:54.366432 28912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:54.359587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:40355 (local address 127.6.61.3:54491)
0812 07:08:54.360205 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:54.360221 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:54.360332 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:54.360684 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:54.360688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:54.360715 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:08:54.361108 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:54.361121 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:54.362463 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:54.362468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:54.364046 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:54.364062 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:54.365541 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:54.365550 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:54.365767 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:54.365802 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 07:08:54.365948 (+ 146us) client_negotiation.cc:240] Negotiation successful
0812 07:08:54.366081 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":1,"thread_start_us":260,"threads_started":1}
I20260812 07:08:54.366895 28692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d8fee246e74aa08e4618615ac5c013" candidate_uuid: "4fdf45109f154799955cae89afaca8fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdb1bb1bac604d2ca2b4a840508b472c" is_pre_election: true
I20260812 07:08:54.367564 28692 raft_consensus.cc:2454] T b9d8fee246e74aa08e4618615ac5c013 P cdb1bb1bac604d2ca2b4a840508b472c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fdf45109f154799955cae89afaca8fe in term 0.
I20260812 07:08:54.367549 28555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d8fee246e74aa08e4618615ac5c013" candidate_uuid: "4fdf45109f154799955cae89afaca8fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ffe0a592e28441a988641a153cf4a79" is_pre_election: true
I20260812 07:08:54.368130 28555 raft_consensus.cc:2454] T b9d8fee246e74aa08e4618615ac5c013 P 2ffe0a592e28441a988641a153cf4a79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fdf45109f154799955cae89afaca8fe in term 0.
I20260812 07:08:54.368368 28761 leader_election.cc:302] T b9d8fee246e74aa08e4618615ac5c013 P 4fdf45109f154799955cae89afaca8fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fdf45109f154799955cae89afaca8fe, cdb1bb1bac604d2ca2b4a840508b472c; no voters:
I20260812 07:08:54.368815 28909 raft_consensus.cc:2781] T b9d8fee246e74aa08e4618615ac5c013 P 4fdf45109f154799955cae89afaca8fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:08:54.368943 28909 raft_consensus.cc:482] T b9d8fee246e74aa08e4618615ac5c013 P 4fdf45109f154799955cae89afaca8fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:08:54.369093 28909 raft_consensus.cc:3037] T b9d8fee246e74aa08e4618615ac5c013 P 4fdf45109f154799955cae89afaca8fe [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:08:54.371434 28909 raft_consensus.cc:504] T b9d8fee246e74aa08e4618615ac5c013 P 4fdf45109f154799955cae89afaca8fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb1bb1bac604d2ca2b4a840508b472c" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 34259 } } peers { permanent_uuid: "4fdf45109f154799955cae89afaca8fe" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 44939 } } peers { permanent_uuid: "2ffe0a592e28441a988641a153cf4a79" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 40355 } }
I20260812 07:08:54.372014 28909 leader_election.cc:288] T b9d8fee246e74aa08e4618615ac5c013 P 4fdf45109f154799955cae89afaca8fe [CANDIDATE]: Term 1 election: Requested vote from peers cdb1bb1bac604d2ca2b4a840508b472c (127.6.61.2:34259), 2ffe0a592e28441a988641a153cf4a79 (127.6.61.1:40355)
I20260812 07:08:54.372583 28692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d8fee246e74aa08e4618615ac5c013" candidate_uuid: "4fdf45109f154799955cae89afaca8fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdb1bb1bac604d2ca2b4a840508b472c"
I20260812 07:08:54.372779 28692 raft_consensus.cc:3037] T b9d8fee246e74aa08e4618615ac5c013 P cdb1bb1bac604d2ca2b4a840508b472c [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:08:54.372746 28555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d8fee246e74aa08e4618615ac5c013" candidate_uuid: "4fdf45109f154799955cae89afaca8fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ffe0a592e28441a988641a153cf4a79"
I20260812 07:08:54.372979 28555 raft_consensus.cc:3037] T b9d8fee246e74aa08e4618615ac5c013 P 2ffe0a592e28441a988641a153cf4a79 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:08:54.375134 28692 raft_consensus.cc:2454] T b9d8fee246e74aa08e4618615ac5c013 P cdb1bb1bac604d2ca2b4a840508b472c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fdf45109f154799955cae89afaca8fe in term 1.
I20260812 07:08:54.375402 28555 raft_consensus.cc:2454] T b9d8fee246e74aa08e4618615ac5c013 P 2ffe0a592e28441a988641a153cf4a79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fdf45109f154799955cae89afaca8fe in term 1.
I20260812 07:08:54.375591 28761 leader_election.cc:302] T b9d8fee246e74aa08e4618615ac5c013 P 4fdf45109f154799955cae89afaca8fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fdf45109f154799955cae89afaca8fe, cdb1bb1bac604d2ca2b4a840508b472c; no voters:
I20260812 07:08:54.375896 28909 raft_consensus.cc:2781] T b9d8fee246e74aa08e4618615ac5c013 P 4fdf45109f154799955cae89afaca8fe [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:08:54.376379 28909 raft_consensus.cc:686] T b9d8fee246e74aa08e4618615ac5c013 P 4fdf45109f154799955cae89afaca8fe [term 1 LEADER]: Becoming Leader. State: Replica: 4fdf45109f154799955cae89afaca8fe, State: Running, Role: LEADER
I20260812 07:08:54.377135 28909 consensus_queue.cc:237] T b9d8fee246e74aa08e4618615ac5c013 P 4fdf45109f154799955cae89afaca8fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb1bb1bac604d2ca2b4a840508b472c" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 34259 } } peers { permanent_uuid: "4fdf45109f154799955cae89afaca8fe" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 44939 } } peers { permanent_uuid: "2ffe0a592e28441a988641a153cf4a79" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 40355 } }
I20260812 07:08:54.381891 28404 catalog_manager.cc:5495] T b9d8fee246e74aa08e4618615ac5c013 P 4fdf45109f154799955cae89afaca8fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fdf45109f154799955cae89afaca8fe (127.6.61.3). New cstate: current_term: 1 leader_uuid: "4fdf45109f154799955cae89afaca8fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb1bb1bac604d2ca2b4a840508b472c" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 34259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fdf45109f154799955cae89afaca8fe" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 44939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ffe0a592e28441a988641a153cf4a79" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 40355 } health_report { overall_health: UNKNOWN } } }
I20260812 07:08:54.455471 28900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:54.449194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55948 (local address 127.6.61.3:44939)
0812 07:08:54.449398 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:54.449403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:54.449937 (+ 534us) server_negotiation.cc:407] Connection header received
0812 07:08:54.450206 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:54.450212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:54.450315 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:54.450468 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:08:54.450960 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:54.451741 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:54.453241 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:54.453697 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:54.453805 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:54.454393 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:08:54.454909 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:08:54.455039 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:54.455218 (+ 179us) server_negotiation.cc:299] Negotiation successful
0812 07:08:54.455258 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 07:08:54.463971 28692 raft_consensus.cc:1261] T b9d8fee246e74aa08e4618615ac5c013 P cdb1bb1bac604d2ca2b4a840508b472c [term 1 FOLLOWER]: Refusing update from remote peer 4fdf45109f154799955cae89afaca8fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:08:54.464006 28555 raft_consensus.cc:1261] T b9d8fee246e74aa08e4618615ac5c013 P 2ffe0a592e28441a988641a153cf4a79 [term 1 FOLLOWER]: Refusing update from remote peer 4fdf45109f154799955cae89afaca8fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:08:54.465006 28913 consensus_queue.cc:1035] T b9d8fee246e74aa08e4618615ac5c013 P 4fdf45109f154799955cae89afaca8fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdb1bb1bac604d2ca2b4a840508b472c" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 34259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:08:54.465454 28909 consensus_queue.cc:1035] T b9d8fee246e74aa08e4618615ac5c013 P 4fdf45109f154799955cae89afaca8fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ffe0a592e28441a988641a153cf4a79" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:08:54.478322 28919 mvcc.cc:205] Tried to move back new op lower bound from 7317579917153284096 to 7317579916813238272. Current Snapshot: MvccSnapshot[applied={T|T < 7317579917153284096}]
I20260812 07:08:54.479409 28921 mvcc.cc:205] Tried to move back new op lower bound from 7317579917153284096 to 7317579916813238272. Current Snapshot: MvccSnapshot[applied={T|T < 7317579917153284096}]
I20260812 07:08:54.492328 28404 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:08:54.492615 28404 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:08:54.496035 28404 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 465d4913edbe45a497ddce45edbed5ef: Sending DeleteTablet for 3 replicas of tablet b9d8fee246e74aa08e4618615ac5c013
I20260812 07:08:54.497217 28670 tablet_service.cc:1514] Processing DeleteTablet for tablet b9d8fee246e74aa08e4618615ac5c013 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:08:54 UTC) from {username='slave'} at 127.0.0.1:53364
I20260812 07:08:54.497346 28811 tablet_service.cc:1514] Processing DeleteTablet for tablet b9d8fee246e74aa08e4618615ac5c013 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:08:54 UTC) from {username='slave'} at 127.0.0.1:55938
I20260812 07:08:54.497643 28532 tablet_service.cc:1514] Processing DeleteTablet for tablet b9d8fee246e74aa08e4618615ac5c013 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:08:54 UTC) from {username='slave'} at 127.0.0.1:59382
I20260812 07:08:54.499311 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 28468
I20260812 07:08:54.507845 28931 tablet_replica.cc:330] T b9d8fee246e74aa08e4618615ac5c013 P 4fdf45109f154799955cae89afaca8fe: stopping tablet replica
I20260812 07:08:54.508474 28931 raft_consensus.cc:2229] T b9d8fee246e74aa08e4618615ac5c013 P 4fdf45109f154799955cae89afaca8fe [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:08:54.510473 28931 raft_consensus.cc:2258] T b9d8fee246e74aa08e4618615ac5c013 P 4fdf45109f154799955cae89afaca8fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:08:54.511457 28930 tablet_replica.cc:330] T b9d8fee246e74aa08e4618615ac5c013 P cdb1bb1bac604d2ca2b4a840508b472c: stopping tablet replica
I20260812 07:08:54.514668 28931 ts_tablet_manager.cc:1905] T b9d8fee246e74aa08e4618615ac5c013 P 4fdf45109f154799955cae89afaca8fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:08:54.516179 28930 raft_consensus.cc:2229] T b9d8fee246e74aa08e4618615ac5c013 P cdb1bb1bac604d2ca2b4a840508b472c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:08:54.517702 28932 tablet_replica.cc:330] T b9d8fee246e74aa08e4618615ac5c013 P 2ffe0a592e28441a988641a153cf4a79: stopping tablet replica
I20260812 07:08:54.518299 28932 raft_consensus.cc:2229] T b9d8fee246e74aa08e4618615ac5c013 P 2ffe0a592e28441a988641a153cf4a79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:08:54.518607 28930 raft_consensus.cc:2258] T b9d8fee246e74aa08e4618615ac5c013 P cdb1bb1bac604d2ca2b4a840508b472c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:08:54.518832 28932 raft_consensus.cc:2258] T b9d8fee246e74aa08e4618615ac5c013 P 2ffe0a592e28441a988641a153cf4a79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:08:54.519367 28902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:54.500545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59396 (local address 127.6.61.1:40355)
0812 07:08:54.500736 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:54.500742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:54.509339 (+ 8597us) server_negotiation.cc:407] Connection header received
0812 07:08:54.509595 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:54.509602 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:54.509716 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:54.509873 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:54.510553 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:54.511788 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:54.514137 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:54.514835 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:54.514943 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:54.516772 (+ 1829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:54.516787 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:54.516817 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:08:54.517651 (+ 834us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:54.517724 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:08:54.517751 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:54.517899 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:54.518864 (+ 965us) server_negotiation.cc:299] Negotiation successful
0812 07:08:54.518995 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 07:08:54.521190 28595 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:08:54.524458 28931 ts_tablet_manager.cc:1918] T b9d8fee246e74aa08e4618615ac5c013 P 4fdf45109f154799955cae89afaca8fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:08:54.524688 28931 log.cc:1196] T b9d8fee246e74aa08e4618615ac5c013 P 4fdf45109f154799955cae89afaca8fe: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-2/wal/wals/b9d8fee246e74aa08e4618615ac5c013
I20260812 07:08:54.525306 28931 ts_tablet_manager.cc:1939] T b9d8fee246e74aa08e4618615ac5c013 P 4fdf45109f154799955cae89afaca8fe: Deleting consensus metadata
I20260812 07:08:54.527616 28388 catalog_manager.cc:4841] TS 4fdf45109f154799955cae89afaca8fe (127.6.61.3:44939): tablet b9d8fee246e74aa08e4618615ac5c013 (table test-table [id=0a29b50de1554043ba652182720627d7]) successfully deleted
I20260812 07:08:54.529699 28930 ts_tablet_manager.cc:1905] T b9d8fee246e74aa08e4618615ac5c013 P cdb1bb1bac604d2ca2b4a840508b472c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:08:54.539218 28930 ts_tablet_manager.cc:1918] T b9d8fee246e74aa08e4618615ac5c013 P cdb1bb1bac604d2ca2b4a840508b472c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:08:54.539590 28930 log.cc:1196] T b9d8fee246e74aa08e4618615ac5c013 P cdb1bb1bac604d2ca2b4a840508b472c: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518208686283-6388-0/minicluster-data/ts-1/wal/wals/b9d8fee246e74aa08e4618615ac5c013
I20260812 07:08:54.540484 28930 ts_tablet_manager.cc:1939] T b9d8fee246e74aa08e4618615ac5c013 P cdb1bb1bac604d2ca2b4a840508b472c: Deleting consensus metadata
I20260812 07:08:54.543165 28387 catalog_manager.cc:4841] TS cdb1bb1bac604d2ca2b4a840508b472c (127.6.61.2:34259): tablet b9d8fee246e74aa08e4618615ac5c013 (table test-table [id=0a29b50de1554043ba652182720627d7]) successfully deleted
I20260812 07:08:54.521334 28932 ts_tablet_manager.cc:1905] T b9d8fee246e74aa08e4618615ac5c013 P 2ffe0a592e28441a988641a153cf4a79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:08:54.623327 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 28468
W20260812 07:08:54.646456 28387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.61.1:40355 (error 108)
I20260812 07:08:54.646688 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 28607
I20260812 07:08:54.648741 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:54.647966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:40355 (local address 127.0.0.1:59410)
0812 07:08:54.648431 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:54.648547 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:40355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":224,"threads_started":1}
W20260812 07:08:54.649014 28387 catalog_manager.cc:4584] TS 2ffe0a592e28441a988641a153cf4a79 (127.6.61.1:40355): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b9d8fee246e74aa08e4618615ac5c013: Network error: Client connection negotiation failed: client connection to 127.6.61.1:40355: connect: Connection refused (error 111)
I20260812 07:08:54.655055 28898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:54.647660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53380 (local address 127.6.61.2:34259)
0812 07:08:54.647867 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:54.647874 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:54.647895 (+ 21us) server_negotiation.cc:407] Connection header received
0812 07:08:54.648011 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:54.648017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:54.648174 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:54.648299 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:54.649027 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:54.649903 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:54.651807 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:54.652216 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:54.652316 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:54.653501 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:54.653510 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:54.653526 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:08:54.654283 (+ 757us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:54.654334 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:08:54.654355 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:54.654455 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:54.654615 (+ 160us) server_negotiation.cc:299] Negotiation successful
0812 07:08:54.654709 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 07:08:54.655812 28735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:08:54.702845 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:54.702270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:40355 (local address 127.0.0.1:59424)
0812 07:08:54.702527 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:54.702639 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:40355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260812 07:08:54.704165 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:54.703672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:40355 (local address 127.0.0.1:59436)
0812 07:08:54.703891 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:54.703987 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:40355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260812 07:08:54.745888 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 28607
I20260812 07:08:54.765293 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:54.764667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:40355 (local address 127.0.0.1:59442)
0812 07:08:54.764943 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:54.765068 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:40355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260812 07:08:54.766785 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:54.766196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:40355 (local address 127.0.0.1:59456)
0812 07:08:54.766453 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:54.766563 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:40355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260812 07:08:54.771076 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 28745
I20260812 07:08:54.780530 28900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:54.772127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55956 (local address 127.6.61.3:44939)
0812 07:08:54.772350 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:54.772356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:54.772382 (+ 26us) server_negotiation.cc:407] Connection header received
0812 07:08:54.772456 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:54.772462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:54.772580 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:54.772720 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:54.773237 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:54.774517 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:54.776125 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:54.776878 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:54.777032 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:54.778525 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:54.778542 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:54.778564 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:08:54.779502 (+ 938us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:54.779583 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:08:54.779621 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:54.779821 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:54.779962 (+ 141us) server_negotiation.cc:299] Negotiation successful
0812 07:08:54.780091 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 07:08:54.781440 28878 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:08:54.847571 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:54.846920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:40355 (local address 127.0.0.1:59464)
0812 07:08:54.847257 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:54.847374 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:40355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260812 07:08:54.848917 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:54.848399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:40355 (local address 127.0.0.1:59474)
0812 07:08:54.848647 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:54.848741 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:40355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260812 07:08:54.907998 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 28745
I20260812 07:08:54.935213 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 28370
I20260812 07:08:54.936467 28436 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:08:55.051479 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:55.050731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:40355 (local address 127.0.0.1:59480)
0812 07:08:55.051102 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:55.051241 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.1:40355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210}
I20260812 07:08:55.052553 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 28370
2026-08-12T07:08:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3374 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:08:55 dist-test-slave-69rg krb5kdc[28942](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:08:55 dist-test-slave-69rg krb5kdc[28942](info): set up 2 sockets
Aug 12 07:08:55 dist-test-slave-69rg krb5kdc[28942](info): commencing operation
krb5kdc: starting...
W20260812 07:08:56.337898 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.223s user 0.001s sys 0.053s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:08:56 dist-test-slave-69rg krb5kdc[28942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:08:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:08:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
I20260812 07:08:56.690232 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:45241
--webserver_interface=127.6.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:43439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:45241
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:08:57.056476 28957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:08:57.056806 28957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:08:57.056959 28957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:08:57.065035 28957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:08:57.065184 28957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:08:57.065306 28957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:08:57.065395 28957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:08:57.065485 28957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:08:57.076133 28957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:43439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:45241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:45241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:08:57.077593 28957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:08:57.079036 28957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:08:57.092124 28965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:57.092262 28962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:57.092188 28963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:08:57.093617 28957 server_base.cc:1034] running on GCE node
I20260812 07:08:57.094545 28957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:08:57.096243 28957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:08:57.097518 28957 hybrid_clock.cc:648] HybridClock initialized: now 1786518537097469 us; error 39 us; skew 500 ppm
I20260812 07:08:57.097982 28957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:08:57 dist-test-slave-69rg krb5kdc[28942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518537, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:08:57.101536 28957 init.cc:377] Logged in from keytab as kudu/127.6.61.62@KRBTEST.COM (short username kudu)
I20260812 07:08:57.103413 28957 webserver.cc:460] Webserver started at http://127.6.61.62:43741/ using document root <none> and password file <none>
I20260812 07:08:57.104179 28957 fs_manager.cc:359] Metadata directory not provided
I20260812 07:08:57.104336 28957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:08:57.104683 28957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:08:57.107996 28957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "1e76b6f5078f469a8ec4b503329b66eb"
format_stamp: "Formatted at 2026-08-12 07:08:57 on dist-test-slave-69rg"
server_key: "1ce1272092440b94e32ad7c6cb8acda9"
server_key_iv: "d5aa8b15a354e0c0bcc7147acc2fd9bc"
server_key_version: "encryptionkey@0"
I20260812 07:08:57.108856 28957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "1e76b6f5078f469a8ec4b503329b66eb"
format_stamp: "Formatted at 2026-08-12 07:08:57 on dist-test-slave-69rg"
server_key: "1ce1272092440b94e32ad7c6cb8acda9"
server_key_iv: "d5aa8b15a354e0c0bcc7147acc2fd9bc"
server_key_version: "encryptionkey@0"
I20260812 07:08:57.114530 28957 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 07:08:57.118444 28973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:57.119916 28957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:08:57.120118 28957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "1e76b6f5078f469a8ec4b503329b66eb"
format_stamp: "Formatted at 2026-08-12 07:08:57 on dist-test-slave-69rg"
server_key: "1ce1272092440b94e32ad7c6cb8acda9"
server_key_iv: "d5aa8b15a354e0c0bcc7147acc2fd9bc"
server_key_version: "encryptionkey@0"
I20260812 07:08:57.120340 28957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:08:57.137502 28957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:08:57.142488 28957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:08:57.142844 28957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:08:57.163627 28957 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:45241
I20260812 07:08:57.163664 29030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:45241 every 8 connection(s)
I20260812 07:08:57.165185 28957 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:08:57.169255 29031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:08:57.175666 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 28957
I20260812 07:08:57.175925 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:08:57.176446 6388 external_mini_cluster.cc:1428] Setting key 36cb0d0ab86e21bec900fdece1a0e783
I20260812 07:08:57.181111 29031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e76b6f5078f469a8ec4b503329b66eb: Bootstrap starting.
I20260812 07:08:57.185110 29031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e76b6f5078f469a8ec4b503329b66eb: Neither blocks nor log segments found. Creating new log.
I20260812 07:08:57.186463 29031 log.cc:824] T 00000000000000000000000000000000 P 1e76b6f5078f469a8ec4b503329b66eb: Log is configured to *not* fsync() on all Append() calls
I20260812 07:08:57.189946 29031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e76b6f5078f469a8ec4b503329b66eb: No bootstrap required, opened a new log
Aug 12 07:08:57 dist-test-slave-69rg krb5kdc[28942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:08:57.200548 29031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e76b6f5078f469a8ec4b503329b66eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e76b6f5078f469a8ec4b503329b66eb" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 45241 } }
I20260812 07:08:57.201067 29031 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e76b6f5078f469a8ec4b503329b66eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:08:57.201185 29031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e76b6f5078f469a8ec4b503329b66eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e76b6f5078f469a8ec4b503329b66eb, State: Initialized, Role: FOLLOWER
I20260812 07:08:57.201951 29031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e76b6f5078f469a8ec4b503329b66eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e76b6f5078f469a8ec4b503329b66eb" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 45241 } }
I20260812 07:08:57.202282 29031 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e76b6f5078f469a8ec4b503329b66eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:08:57.202468 29031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e76b6f5078f469a8ec4b503329b66eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:08:57.202687 29031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e76b6f5078f469a8ec4b503329b66eb [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:08:57.204228 29034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:57.178706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45646 (local address 127.6.61.62:45241)
0812 07:08:57.179537 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:57.179555 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:57.179607 (+ 52us) server_negotiation.cc:407] Connection header received
0812 07:08:57.182122 (+ 2515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:57.182151 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:57.182645 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:57.183162 (+ 517us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:57.184557 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:57.186648 (+ 2091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:57.188403 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:57.189476 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:57.189675 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:57.194319 (+ 4644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:57.194336 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:57.194359 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:08:57.194416 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:57.198959 (+ 4543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:57.199782 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:57.199792 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:57.199801 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:57.200015 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:57.200567 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:57.200576 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:57.200581 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:57.201373 (+ 792us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:57.201684 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:57.202319 (+ 635us) server_negotiation.cc:299] Negotiation successful
0812 07:08:57.202813 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":360,"threads_started":1}
I20260812 07:08:57.205662 29031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e76b6f5078f469a8ec4b503329b66eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e76b6f5078f469a8ec4b503329b66eb" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 45241 } }
I20260812 07:08:57.206193 29031 leader_election.cc:302] T 00000000000000000000000000000000 P 1e76b6f5078f469a8ec4b503329b66eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e76b6f5078f469a8ec4b503329b66eb; no voters:
I20260812 07:08:57.206864 29031 leader_election.cc:288] T 00000000000000000000000000000000 P 1e76b6f5078f469a8ec4b503329b66eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:08:57.207082 29036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e76b6f5078f469a8ec4b503329b66eb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:08:57.208488 29036 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e76b6f5078f469a8ec4b503329b66eb [term 1 LEADER]: Becoming Leader. State: Replica: 1e76b6f5078f469a8ec4b503329b66eb, State: Running, Role: LEADER
I20260812 07:08:57.209054 29036 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e76b6f5078f469a8ec4b503329b66eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e76b6f5078f469a8ec4b503329b66eb" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 45241 } }
I20260812 07:08:57.209350 29031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e76b6f5078f469a8ec4b503329b66eb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:08:57.212520 29038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e76b6f5078f469a8ec4b503329b66eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e76b6f5078f469a8ec4b503329b66eb. Latest consensus state: current_term: 1 leader_uuid: "1e76b6f5078f469a8ec4b503329b66eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e76b6f5078f469a8ec4b503329b66eb" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 45241 } } }
I20260812 07:08:57.212990 29038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e76b6f5078f469a8ec4b503329b66eb [sys.catalog]: This master's current role is: LEADER
I20260812 07:08:57.213603 29037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e76b6f5078f469a8ec4b503329b66eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e76b6f5078f469a8ec4b503329b66eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e76b6f5078f469a8ec4b503329b66eb" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 45241 } } }
I20260812 07:08:57.213996 29037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e76b6f5078f469a8ec4b503329b66eb [sys.catalog]: This master's current role is: LEADER
I20260812 07:08:57.214965 29043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:08:57.221657 29043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:08:57.233137 29043 catalog_manager.cc:1347] Generated new cluster ID: 7e0de4f0ffd64714b1200792000fa2de
I20260812 07:08:57.233271 29043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:08:57.249581 29043 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:08:57.251539 29043 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:08:57.262269 29043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e76b6f5078f469a8ec4b503329b66eb: Generated new TSK 0
I20260812 07:08:57.263365 29043 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
Entry for principal kudu/127.6.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab.
I20260812 07:08:57.340178 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.1:0
--local_ip_for_outbound_sockets=127.6.61.1
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:45241
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:43439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:08:57.713302 29058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:08:57.713685 29058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:08:57.713946 29058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:08:57.722332 29058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:08:57.722494 29058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:08:57.722671 29058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.1
I20260812 07:08:57.733368 29058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:43439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.1.keytab
--principal=kudu/127.6.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:45241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.1
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:08:57.734805 29058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:08:57.736145 29058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:08:57.750399 29064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:57.750519 29065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:57.750531 29067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:08:57.752539 29058 server_base.cc:1034] running on GCE node
I20260812 07:08:57.753333 29058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:08:57.754492 29058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:08:57.755777 29058 hybrid_clock.cc:648] HybridClock initialized: now 1786518537755718 us; error 71 us; skew 500 ppm
I20260812 07:08:57.756168 29058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:08:57 dist-test-slave-69rg krb5kdc[28942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518537, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:08:57.764720 29058 init.cc:377] Logged in from keytab as kudu/127.6.61.1@KRBTEST.COM (short username kudu)
I20260812 07:08:57.766633 29058 webserver.cc:460] Webserver started at http://127.6.61.1:37083/ using document root <none> and password file <none>
I20260812 07:08:57.767499 29058 fs_manager.cc:359] Metadata directory not provided
I20260812 07:08:57.767673 29058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:08:57.768044 29058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:08:57.771610 29058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-0/data/instance:
uuid: "efc3a6eacf2e40f59b33a56e4ce46446"
format_stamp: "Formatted at 2026-08-12 07:08:57 on dist-test-slave-69rg"
server_key: "e0fe1528d1eeb4c4a60e5480f7c299f5"
server_key_iv: "b159c15855559c17a50d65612bd14f77"
server_key_version: "encryptionkey@0"
I20260812 07:08:57.772605 29058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance:
uuid: "efc3a6eacf2e40f59b33a56e4ce46446"
format_stamp: "Formatted at 2026-08-12 07:08:57 on dist-test-slave-69rg"
server_key: "e0fe1528d1eeb4c4a60e5480f7c299f5"
server_key_iv: "b159c15855559c17a50d65612bd14f77"
server_key_version: "encryptionkey@0"
I20260812 07:08:57.778450 29058 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260812 07:08:57.782347 29075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:57.783977 29058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 07:08:57.784196 29058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-0/wal
uuid: "efc3a6eacf2e40f59b33a56e4ce46446"
format_stamp: "Formatted at 2026-08-12 07:08:57 on dist-test-slave-69rg"
server_key: "e0fe1528d1eeb4c4a60e5480f7c299f5"
server_key_iv: "b159c15855559c17a50d65612bd14f77"
server_key_version: "encryptionkey@0"
I20260812 07:08:57.784421 29058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:08:57.803800 29058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:08:57.807917 29058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:08:57.808254 29058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:08:57.809434 29058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:08:57.811448 29058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:08:57.811599 29058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260812 07:08:57.811785 29058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:08:57.811856 29058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:57.847111 29058 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.1:42993
I20260812 07:08:57.847203 29197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.1:42993 every 8 connection(s)
I20260812 07:08:57.848389 29058 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
I20260812 07:08:57.857420 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 29058
I20260812 07:08:57.857743 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance
I20260812 07:08:57.858306 6388 external_mini_cluster.cc:1428] Setting key cad43f02fbc49eee8c247eaadde8b3df
Aug 12 07:08:57 dist-test-slave-69rg krb5kdc[28942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518537, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.1@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:08:57.871017 29200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:57.851275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:57751 (local address 127.6.61.62:45241)
0812 07:08:57.851797 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:57.851803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:57.852801 (+ 998us) server_negotiation.cc:407] Connection header received
0812 07:08:57.854920 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:57.854924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:57.854994 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:57.855116 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:57.856765 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:57.857598 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:57.859612 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:57.860243 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:57.860661 (+ 418us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:57.864946 (+ 4285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:57.864952 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:57.864956 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:08:57.864992 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:57.867733 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:57.868499 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:57.868512 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:57.868515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:57.868637 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:57.869099 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:57.869108 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:57.869111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:57.869399 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:57.869564 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:57.870379 (+ 815us) server_negotiation.cc:299] Negotiation successful
0812 07:08:57.870627 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":246,"threads_started":1}
I20260812 07:08:57.872020 29201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:57.851759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:45241 (local address 127.6.61.1:57751)
0812 07:08:57.852604 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:57.852659 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:57.854623 (+ 1964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:57.855372 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:57.855380 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:57.855940 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:08:57.856565 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:57.856584 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:57.857837 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:57.857844 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:57.859389 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:57.859403 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:57.860984 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:57.860990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:57.861220 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:57.862060 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:08:57.862089 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:08:57.864672 (+ 2583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:08:57.867886 (+ 3214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:08:57.867896 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:08:57.867914 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:08:57.868358 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:08:57.868759 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:08:57.868764 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:08:57.868767 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:08:57.868970 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:08:57.869542 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:08:57.869550 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:08:57.869976 (+ 426us) client_negotiation.cc:769] Sending connection context
0812 07:08:57.870261 (+ 285us) client_negotiation.cc:240] Negotiation successful
0812 07:08:57.870839 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":361,"threads_started":1}
I20260812 07:08:57.873796 29198 heartbeater.cc:347] Connected to a master server at 127.6.61.62:45241
I20260812 07:08:57.874158 29198 heartbeater.cc:464] Registering TS with master...
I20260812 07:08:57.874987 29198 heartbeater.cc:511] Master 127.6.61.62:45241 requested a full tablet report, sending...
I20260812 07:08:57.877408 28995 ts_manager.cc:194] Registered new tserver with Master: efc3a6eacf2e40f59b33a56e4ce46446 (127.6.61.1:42993)
I20260812 07:08:57.879145 28995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.1@KRBTEST.COM'} at 127.6.61.1:57751
WARNING: no policy specified for kudu/127.6.61.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
Entry for principal kudu/127.6.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab.
I20260812 07:08:57.935343 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.2:0
--local_ip_for_outbound_sockets=127.6.61.2
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:45241
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:43439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:08:58.302546 29204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:08:58.302860 29204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:08:58.302955 29204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:08:58.310702 29204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:08:58.310833 29204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:08:58.310950 29204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.2
I20260812 07:08:58.321260 29204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:43439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.2.keytab
--principal=kudu/127.6.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:45241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.2
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:08:58.322633 29204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:08:58.323968 29204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:08:58.339597 29209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:58.340006 29210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:58.340337 29213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:08:58.343591 29204 server_base.cc:1034] running on GCE node
I20260812 07:08:58.344375 29204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:08:58.345472 29204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:08:58.346892 29204 hybrid_clock.cc:648] HybridClock initialized: now 1786518538346821 us; error 47 us; skew 500 ppm
I20260812 07:08:58.347347 29204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:08:58 dist-test-slave-69rg krb5kdc[28942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518538, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:08:58.351397 29204 init.cc:377] Logged in from keytab as kudu/127.6.61.2@KRBTEST.COM (short username kudu)
I20260812 07:08:58.353314 29204 webserver.cc:460] Webserver started at http://127.6.61.2:34679/ using document root <none> and password file <none>
I20260812 07:08:58.354192 29204 fs_manager.cc:359] Metadata directory not provided
I20260812 07:08:58.354344 29204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:08:58.354681 29204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:08:58.357743 29204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-1/data/instance:
uuid: "aef17fb49def4c8d9ca13ac09d765375"
format_stamp: "Formatted at 2026-08-12 07:08:58 on dist-test-slave-69rg"
server_key: "9351321038ea7e6cf70c28070065dee7"
server_key_iv: "50da374d7ad934b388b5ed76cb06a316"
server_key_version: "encryptionkey@0"
I20260812 07:08:58.358677 29204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance:
uuid: "aef17fb49def4c8d9ca13ac09d765375"
format_stamp: "Formatted at 2026-08-12 07:08:58 on dist-test-slave-69rg"
server_key: "9351321038ea7e6cf70c28070065dee7"
server_key_iv: "50da374d7ad934b388b5ed76cb06a316"
server_key_version: "encryptionkey@0"
I20260812 07:08:58.364481 29204 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 07:08:58.368522 29220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:58.370517 29204 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260812 07:08:58.370759 29204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-1/wal
uuid: "aef17fb49def4c8d9ca13ac09d765375"
format_stamp: "Formatted at 2026-08-12 07:08:58 on dist-test-slave-69rg"
server_key: "9351321038ea7e6cf70c28070065dee7"
server_key_iv: "50da374d7ad934b388b5ed76cb06a316"
server_key_version: "encryptionkey@0"
I20260812 07:08:58.371023 29204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:08:58.405346 29204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:08:58.410396 29204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:08:58.410750 29204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:08:58.411859 29204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:08:58.413354 29204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:08:58.413465 29204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:58.413626 29204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:08:58.413708 29204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:58.452601 29204 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.2:41431
I20260812 07:08:58.452656 29340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.2:41431 every 8 connection(s)
I20260812 07:08:58.454005 29204 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
I20260812 07:08:58.461771 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 29204
I20260812 07:08:58.462169 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance
I20260812 07:08:58.462765 6388 external_mini_cluster.cc:1428] Setting key b97b183a12c05446dd26022d2a4ff4cd
Aug 12 07:08:58 dist-test-slave-69rg krb5kdc[28942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518538, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.2@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:08:58.478670 29343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:58.456933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:41349 (local address 127.6.61.62:45241)
0812 07:08:58.457492 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:58.457498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:58.458536 (+ 1038us) server_negotiation.cc:407] Connection header received
0812 07:08:58.460665 (+ 2129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:58.460671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:58.460789 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:58.460930 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:58.463171 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:58.464164 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:58.466220 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:58.466959 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:58.467116 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:58.470912 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:58.470921 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:58.470928 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:08:58.470976 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:58.474842 (+ 3866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:58.475701 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:58.475709 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:58.475712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:58.475880 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:58.476460 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:58.476467 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:58.476473 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:58.476884 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:58.477045 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:58.477876 (+ 831us) server_negotiation.cc:299] Negotiation successful
0812 07:08:58.478162 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":292,"threads_started":1}
I20260812 07:08:58.479439 29344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:58.457344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:45241 (local address 127.6.61.2:41349)
0812 07:08:58.458350 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:58.458398 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:58.460342 (+ 1944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:58.461260 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:58.461270 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:58.462001 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:08:58.462776 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:58.462797 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:58.464369 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:58.464377 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:58.466049 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:58.466064 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:58.467201 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:58.467207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:58.467444 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:58.468221 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:08:58.468251 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:08:58.470620 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:08:58.475023 (+ 4403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:08:58.475032 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:08:58.475048 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:08:58.475558 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:08:58.476045 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:08:58.476054 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:08:58.476060 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:08:58.476297 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:08:58.477051 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:08:58.477058 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:08:58.477449 (+ 391us) client_negotiation.cc:769] Sending connection context
0812 07:08:58.477699 (+ 250us) client_negotiation.cc:240] Negotiation successful
0812 07:08:58.478209 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":337,"threads_started":1}
I20260812 07:08:58.481185 29341 heartbeater.cc:347] Connected to a master server at 127.6.61.62:45241
I20260812 07:08:58.481550 29341 heartbeater.cc:464] Registering TS with master...
I20260812 07:08:58.482519 29341 heartbeater.cc:511] Master 127.6.61.62:45241 requested a full tablet report, sending...
I20260812 07:08:58.484380 28995 ts_manager.cc:194] Registered new tserver with Master: aef17fb49def4c8d9ca13ac09d765375 (127.6.61.2:41431)
I20260812 07:08:58.485289 28995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.2@KRBTEST.COM'} at 127.6.61.2:41349
WARNING: no policy specified for kudu/127.6.61.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
Entry for principal kudu/127.6.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab.
I20260812 07:08:58.546360 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.3:0
--local_ip_for_outbound_sockets=127.6.61.3
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:45241
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:43439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:08:58.883116 29198 heartbeater.cc:503] Master 127.6.61.62:45241 was elected leader, sending a full tablet report...
W20260812 07:08:58.925407 29347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:08:58.925781 29347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:08:58.925971 29347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:08:58.933861 29347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:08:58.934036 29347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:08:58.934221 29347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.3
I20260812 07:08:58.944749 29347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:43439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.3.keytab
--principal=kudu/127.6.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:45241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.3
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:08:58.946362 29347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:08:58.947796 29347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:08:58.963995 29353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:58.963995 29352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:08:58.964061 29355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:08:58.965173 29347 server_base.cc:1034] running on GCE node
I20260812 07:08:58.966018 29347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:08:58.967082 29347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:08:58.968370 29347 hybrid_clock.cc:648] HybridClock initialized: now 1786518538968310 us; error 38 us; skew 500 ppm
I20260812 07:08:58.968786 29347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:08:58 dist-test-slave-69rg krb5kdc[28942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518538, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:08:58.972730 29347 init.cc:377] Logged in from keytab as kudu/127.6.61.3@KRBTEST.COM (short username kudu)
I20260812 07:08:58.974705 29347 webserver.cc:460] Webserver started at http://127.6.61.3:41767/ using document root <none> and password file <none>
I20260812 07:08:58.975476 29347 fs_manager.cc:359] Metadata directory not provided
I20260812 07:08:58.975642 29347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:08:58.976012 29347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:08:58.979220 29347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-2/data/instance:
uuid: "f99020fc71a943a6af10cb3407cb87b6"
format_stamp: "Formatted at 2026-08-12 07:08:58 on dist-test-slave-69rg"
server_key: "712e6f0dfbde57b0fb0a34cd0bb975c4"
server_key_iv: "5807c69099254e2fafc17fa0020e1f1e"
server_key_version: "encryptionkey@0"
I20260812 07:08:58.980187 29347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance:
uuid: "f99020fc71a943a6af10cb3407cb87b6"
format_stamp: "Formatted at 2026-08-12 07:08:58 on dist-test-slave-69rg"
server_key: "712e6f0dfbde57b0fb0a34cd0bb975c4"
server_key_iv: "5807c69099254e2fafc17fa0020e1f1e"
server_key_version: "encryptionkey@0"
I20260812 07:08:58.986447 29347 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260812 07:08:58.990670 29363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:58.992380 29347 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 07:08:58.992595 29347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-2/wal
uuid: "f99020fc71a943a6af10cb3407cb87b6"
format_stamp: "Formatted at 2026-08-12 07:08:58 on dist-test-slave-69rg"
server_key: "712e6f0dfbde57b0fb0a34cd0bb975c4"
server_key_iv: "5807c69099254e2fafc17fa0020e1f1e"
server_key_version: "encryptionkey@0"
I20260812 07:08:58.992878 29347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:08:59.010165 29347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:08:59.016111 29347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:08:59.016589 29347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:08:59.017789 29347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:08:59.019368 29347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:08:59.019479 29347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:59.019639 29347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:08:59.019716 29347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:08:59.059926 29347 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.3:43609
I20260812 07:08:59.059952 29489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.3:43609 every 8 connection(s)
I20260812 07:08:59.061501 29347 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
I20260812 07:08:59.063651 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 29347
I20260812 07:08:59.063884 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance
I20260812 07:08:59.064612 6388 external_mini_cluster.cc:1428] Setting key 5b044527d1f47d9ad1201ee721935fee
Aug 12 07:08:59 dist-test-slave-69rg krb5kdc[28942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518538, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.3@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:08:59.086545 29493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:59.064972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:45637 (local address 127.6.61.62:45241)
0812 07:08:59.065562 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:59.065568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:59.067051 (+ 1483us) server_negotiation.cc:407] Connection header received
0812 07:08:59.069062 (+ 2011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:59.069067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:59.069157 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:59.069266 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:59.071159 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:59.072063 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:59.073760 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:59.074287 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:59.074409 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:59.078924 (+ 4515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:59.078930 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:59.078933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:08:59.078976 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:59.082566 (+ 3590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:59.083478 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:59.083483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:59.083485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:59.083626 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:59.084378 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:59.084382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:59.084384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:59.084742 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:59.084927 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:59.085960 (+ 1033us) server_negotiation.cc:299] Negotiation successful
0812 07:08:59.086213 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":301,"threads_started":1}
I20260812 07:08:59.087523 29494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:59.065905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:45241 (local address 127.6.61.3:45637)
0812 07:08:59.066869 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:59.066921 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:59.068776 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:59.069593 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:59.069603 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:59.070335 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:08:59.070954 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:59.070973 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:59.072248 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:59.072253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:59.073586 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:59.073601 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:59.074575 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:59.074582 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:59.074993 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:59.075824 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:08:59.075851 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:08:59.078591 (+ 2740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:08:59.082764 (+ 4173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:08:59.082775 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:08:59.082791 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:08:59.083302 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:08:59.083809 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:08:59.083814 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:08:59.083817 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:08:59.084114 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:08:59.084971 (+ 857us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:08:59.084979 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:08:59.085381 (+ 402us) client_negotiation.cc:769] Sending connection context
0812 07:08:59.085669 (+ 288us) client_negotiation.cc:240] Negotiation successful
0812 07:08:59.086210 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":416,"threads_started":1}
I20260812 07:08:59.089545 29491 heartbeater.cc:347] Connected to a master server at 127.6.61.62:45241
I20260812 07:08:59.090001 29491 heartbeater.cc:464] Registering TS with master...
I20260812 07:08:59.090896 29491 heartbeater.cc:511] Master 127.6.61.62:45241 requested a full tablet report, sending...
I20260812 07:08:59.092797 28995 ts_manager.cc:194] Registered new tserver with Master: f99020fc71a943a6af10cb3407cb87b6 (127.6.61.3:43609)
I20260812 07:08:59.093869 28995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.61.3@KRBTEST.COM'} at 127.6.61.3:45637
I20260812 07:08:59.094677 6388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 07:08:59.117183 29493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:59.100287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45658 (local address 127.6.61.62:45241)
0812 07:08:59.100493 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:59.100498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:59.100891 (+ 393us) server_negotiation.cc:407] Connection header received
0812 07:08:59.101092 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:59.101097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:59.101186 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:59.101305 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:59.103071 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:59.104003 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:59.105548 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:59.106123 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:59.106275 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:59.110186 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:59.110192 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:59.110196 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:08:59.110259 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:59.114028 (+ 3769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:59.114919 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:59.114923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:59.114926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:59.115093 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:59.115610 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:59.115614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:59.115617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:59.116035 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:59.116269 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:59.116703 (+ 434us) server_negotiation.cc:299] Negotiation successful
0812 07:08:59.116931 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 07:08:59.122076 28995 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:08:59.130501 28995 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:08:59.157220 29505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:59.144335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:41431 (local address 127.0.0.1:60136)
0812 07:08:59.147277 (+ 2942us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:59.147309 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:59.147472 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:59.149002 (+ 1530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:59.149009 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:59.149037 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:08:59.149505 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:59.149519 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:59.151510 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:59.151518 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:59.153807 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:59.153848 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:59.155954 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:59.155964 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:59.156220 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:59.156292 (+ 72us) client_negotiation.cc:769] Sending connection context
0812 07:08:59.156448 (+ 156us) client_negotiation.cc:240] Negotiation successful
0812 07:08:59.156682 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2785,"mutex_wait_us":27,"thread_start_us":2583,"threads_started":1}
I20260812 07:08:59.157382 29509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:59.148058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52336 (local address 127.6.61.1:42993)
0812 07:08:59.148747 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:59.148756 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:59.148783 (+ 27us) server_negotiation.cc:407] Connection header received
0812 07:08:59.149065 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:59.149072 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:59.149284 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:59.149467 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:08:59.150325 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:59.152038 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:59.154742 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:59.156233 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:59.156395 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:59.156629 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:59.156861 (+ 232us) server_negotiation.cc:299] Negotiation successful
0812 07:08:59.157035 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":364,"threads_started":1}
I20260812 07:08:59.157684 29508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:59.147312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:42993 (local address 127.0.0.1:52336)
0812 07:08:59.148128 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:59.148147 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:59.148867 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:59.149766 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:59.149770 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:59.149794 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:08:59.150166 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:59.150177 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:59.152251 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:59.152256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:59.154520 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:59.154535 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:59.156460 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:59.156467 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:59.156648 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:59.156684 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 07:08:59.156809 (+ 125us) client_negotiation.cc:240] Negotiation successful
0812 07:08:59.156968 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":9,"thread_start_us":954,"threads_started":1}
I20260812 07:08:59.158227 29507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:59.144742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60136 (local address 127.6.61.2:41431)
0812 07:08:59.146746 (+ 2004us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:59.146756 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:59.147441 (+ 685us) server_negotiation.cc:407] Connection header received
0812 07:08:59.148524 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:59.148532 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:59.148763 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:59.149723 (+ 960us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:08:59.149973 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:59.151329 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:59.154047 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:59.155688 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:59.155844 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:59.156072 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:59.157630 (+ 1558us) server_negotiation.cc:299] Negotiation successful
0812 07:08:59.157863 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1837,"thread_start_us":1685,"threads_started":1}
I20260812 07:08:59.158902 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:59.144121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:43609 (local address 127.0.0.1:41478)
0812 07:08:59.145190 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:59.145255 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:59.145505 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:59.147649 (+ 2144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:59.147656 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:59.147689 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:08:59.148231 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:59.148251 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:59.150642 (+ 2391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:59.150650 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:59.154500 (+ 3850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:59.154514 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:59.156949 (+ 2435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:59.156955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:59.157152 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:59.157199 (+ 47us) client_negotiation.cc:769] Sending connection context
0812 07:08:59.157293 (+ 94us) client_negotiation.cc:240] Negotiation successful
0812 07:08:59.158537 (+ 1244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":56,"thread_start_us":494,"threads_started":1}
I20260812 07:08:59.159883 29506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:59.144472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41478 (local address 127.6.61.3:43609)
0812 07:08:59.146295 (+ 1823us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:59.146306 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:59.146351 (+ 45us) server_negotiation.cc:407] Connection header received
0812 07:08:59.146474 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:59.146494 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:59.147394 (+ 900us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:59.147718 (+ 324us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:08:59.148717 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:59.150492 (+ 1775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:59.154702 (+ 4210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:59.155737 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:59.159176 (+ 3439us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:59.159350 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:59.159409 (+ 59us) server_negotiation.cc:299] Negotiation successful
0812 07:08:59.159536 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1644,"thread_start_us":1456,"threads_started":1}
I20260812 07:08:59.162359 29271 tablet_service.cc:1467] Processing CreateTablet for tablet 4454aeb47aef4d7d870d64caeb56bf65 (DEFAULT_TABLE table=test-table [id=9c967e25d4394329a376889e312c74f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:08:59.162658 29128 tablet_service.cc:1467] Processing CreateTablet for tablet 4454aeb47aef4d7d870d64caeb56bf65 (DEFAULT_TABLE table=test-table [id=9c967e25d4394329a376889e312c74f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:08:59.163717 29414 tablet_service.cc:1467] Processing CreateTablet for tablet 4454aeb47aef4d7d870d64caeb56bf65 (DEFAULT_TABLE table=test-table [id=9c967e25d4394329a376889e312c74f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:08:59.164743 29271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4454aeb47aef4d7d870d64caeb56bf65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:08:59.164873 29128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4454aeb47aef4d7d870d64caeb56bf65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:08:59.165015 29414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4454aeb47aef4d7d870d64caeb56bf65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:08:59.190706 29512 tablet_bootstrap.cc:492] T 4454aeb47aef4d7d870d64caeb56bf65 P aef17fb49def4c8d9ca13ac09d765375: Bootstrap starting.
I20260812 07:08:59.191272 29511 tablet_bootstrap.cc:492] T 4454aeb47aef4d7d870d64caeb56bf65 P f99020fc71a943a6af10cb3407cb87b6: Bootstrap starting.
I20260812 07:08:59.194120 29510 tablet_bootstrap.cc:492] T 4454aeb47aef4d7d870d64caeb56bf65 P efc3a6eacf2e40f59b33a56e4ce46446: Bootstrap starting.
I20260812 07:08:59.194196 29512 tablet_bootstrap.cc:654] T 4454aeb47aef4d7d870d64caeb56bf65 P aef17fb49def4c8d9ca13ac09d765375: Neither blocks nor log segments found. Creating new log.
I20260812 07:08:59.194545 29511 tablet_bootstrap.cc:654] T 4454aeb47aef4d7d870d64caeb56bf65 P f99020fc71a943a6af10cb3407cb87b6: Neither blocks nor log segments found. Creating new log.
I20260812 07:08:59.195698 29511 log.cc:824] T 4454aeb47aef4d7d870d64caeb56bf65 P f99020fc71a943a6af10cb3407cb87b6: Log is configured to *not* fsync() on all Append() calls
I20260812 07:08:59.195721 29512 log.cc:824] T 4454aeb47aef4d7d870d64caeb56bf65 P aef17fb49def4c8d9ca13ac09d765375: Log is configured to *not* fsync() on all Append() calls
I20260812 07:08:59.199855 29511 tablet_bootstrap.cc:492] T 4454aeb47aef4d7d870d64caeb56bf65 P f99020fc71a943a6af10cb3407cb87b6: No bootstrap required, opened a new log
I20260812 07:08:59.200054 29510 tablet_bootstrap.cc:654] T 4454aeb47aef4d7d870d64caeb56bf65 P efc3a6eacf2e40f59b33a56e4ce46446: Neither blocks nor log segments found. Creating new log.
I20260812 07:08:59.200237 29511 ts_tablet_manager.cc:1397] T 4454aeb47aef4d7d870d64caeb56bf65 P f99020fc71a943a6af10cb3407cb87b6: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260812 07:08:59.200598 29512 tablet_bootstrap.cc:492] T 4454aeb47aef4d7d870d64caeb56bf65 P aef17fb49def4c8d9ca13ac09d765375: No bootstrap required, opened a new log
I20260812 07:08:59.200893 29512 ts_tablet_manager.cc:1397] T 4454aeb47aef4d7d870d64caeb56bf65 P aef17fb49def4c8d9ca13ac09d765375: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20260812 07:08:59.201474 29510 log.cc:824] T 4454aeb47aef4d7d870d64caeb56bf65 P efc3a6eacf2e40f59b33a56e4ce46446: Log is configured to *not* fsync() on all Append() calls
I20260812 07:08:59.204857 29510 tablet_bootstrap.cc:492] T 4454aeb47aef4d7d870d64caeb56bf65 P efc3a6eacf2e40f59b33a56e4ce46446: No bootstrap required, opened a new log
I20260812 07:08:59.205147 29510 ts_tablet_manager.cc:1397] T 4454aeb47aef4d7d870d64caeb56bf65 P efc3a6eacf2e40f59b33a56e4ce46446: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.000s
I20260812 07:08:59.208957 29511 raft_consensus.cc:348] T 4454aeb47aef4d7d870d64caeb56bf65 P f99020fc71a943a6af10cb3407cb87b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99020fc71a943a6af10cb3407cb87b6" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 43609 } } peers { permanent_uuid: "aef17fb49def4c8d9ca13ac09d765375" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 41431 } } peers { permanent_uuid: "efc3a6eacf2e40f59b33a56e4ce46446" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42993 } }
I20260812 07:08:59.209420 29511 raft_consensus.cc:374] T 4454aeb47aef4d7d870d64caeb56bf65 P f99020fc71a943a6af10cb3407cb87b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:08:59.209563 29511 raft_consensus.cc:729] T 4454aeb47aef4d7d870d64caeb56bf65 P f99020fc71a943a6af10cb3407cb87b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f99020fc71a943a6af10cb3407cb87b6, State: Initialized, Role: FOLLOWER
I20260812 07:08:59.210374 29511 consensus_queue.cc:260] T 4454aeb47aef4d7d870d64caeb56bf65 P f99020fc71a943a6af10cb3407cb87b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99020fc71a943a6af10cb3407cb87b6" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 43609 } } peers { permanent_uuid: "aef17fb49def4c8d9ca13ac09d765375" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 41431 } } peers { permanent_uuid: "efc3a6eacf2e40f59b33a56e4ce46446" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42993 } }
I20260812 07:08:59.211905 29512 raft_consensus.cc:348] T 4454aeb47aef4d7d870d64caeb56bf65 P aef17fb49def4c8d9ca13ac09d765375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99020fc71a943a6af10cb3407cb87b6" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 43609 } } peers { permanent_uuid: "aef17fb49def4c8d9ca13ac09d765375" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 41431 } } peers { permanent_uuid: "efc3a6eacf2e40f59b33a56e4ce46446" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42993 } }
I20260812 07:08:59.212375 29512 raft_consensus.cc:374] T 4454aeb47aef4d7d870d64caeb56bf65 P aef17fb49def4c8d9ca13ac09d765375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:08:59.212483 29512 raft_consensus.cc:729] T 4454aeb47aef4d7d870d64caeb56bf65 P aef17fb49def4c8d9ca13ac09d765375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aef17fb49def4c8d9ca13ac09d765375, State: Initialized, Role: FOLLOWER
I20260812 07:08:59.213153 29512 consensus_queue.cc:260] T 4454aeb47aef4d7d870d64caeb56bf65 P aef17fb49def4c8d9ca13ac09d765375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99020fc71a943a6af10cb3407cb87b6" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 43609 } } peers { permanent_uuid: "aef17fb49def4c8d9ca13ac09d765375" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 41431 } } peers { permanent_uuid: "efc3a6eacf2e40f59b33a56e4ce46446" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42993 } }
I20260812 07:08:59.214713 29341 heartbeater.cc:503] Master 127.6.61.62:45241 was elected leader, sending a full tablet report...
I20260812 07:08:59.215265 29512 ts_tablet_manager.cc:1428] T 4454aeb47aef4d7d870d64caeb56bf65 P aef17fb49def4c8d9ca13ac09d765375: Time spent starting tablet: real 0.014s user 0.007s sys 0.008s
I20260812 07:08:59.216048 29510 raft_consensus.cc:348] T 4454aeb47aef4d7d870d64caeb56bf65 P efc3a6eacf2e40f59b33a56e4ce46446 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99020fc71a943a6af10cb3407cb87b6" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 43609 } } peers { permanent_uuid: "aef17fb49def4c8d9ca13ac09d765375" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 41431 } } peers { permanent_uuid: "efc3a6eacf2e40f59b33a56e4ce46446" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42993 } }
I20260812 07:08:59.216598 29510 raft_consensus.cc:374] T 4454aeb47aef4d7d870d64caeb56bf65 P efc3a6eacf2e40f59b33a56e4ce46446 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:08:59.216775 29510 raft_consensus.cc:729] T 4454aeb47aef4d7d870d64caeb56bf65 P efc3a6eacf2e40f59b33a56e4ce46446 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efc3a6eacf2e40f59b33a56e4ce46446, State: Initialized, Role: FOLLOWER
I20260812 07:08:59.217324 29491 heartbeater.cc:503] Master 127.6.61.62:45241 was elected leader, sending a full tablet report...
I20260812 07:08:59.217644 29510 consensus_queue.cc:260] T 4454aeb47aef4d7d870d64caeb56bf65 P efc3a6eacf2e40f59b33a56e4ce46446 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99020fc71a943a6af10cb3407cb87b6" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 43609 } } peers { permanent_uuid: "aef17fb49def4c8d9ca13ac09d765375" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 41431 } } peers { permanent_uuid: "efc3a6eacf2e40f59b33a56e4ce46446" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42993 } }
I20260812 07:08:59.218101 29511 ts_tablet_manager.cc:1428] T 4454aeb47aef4d7d870d64caeb56bf65 P f99020fc71a943a6af10cb3407cb87b6: Time spent starting tablet: real 0.018s user 0.010s sys 0.009s
I20260812 07:08:59.220157 29510 ts_tablet_manager.cc:1428] T 4454aeb47aef4d7d870d64caeb56bf65 P efc3a6eacf2e40f59b33a56e4ce46446: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
W20260812 07:08:59.313519 29492 tablet.cc:2369] T 4454aeb47aef4d7d870d64caeb56bf65 P f99020fc71a943a6af10cb3407cb87b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 07:08:59.351240 29199 tablet.cc:2369] T 4454aeb47aef4d7d870d64caeb56bf65 P efc3a6eacf2e40f59b33a56e4ce46446: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 07:08:59.456372 29342 tablet.cc:2369] T 4454aeb47aef4d7d870d64caeb56bf65 P aef17fb49def4c8d9ca13ac09d765375: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:08:59.464396 29516 raft_consensus.cc:482] T 4454aeb47aef4d7d870d64caeb56bf65 P f99020fc71a943a6af10cb3407cb87b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:08:59.464769 29516 raft_consensus.cc:504] T 4454aeb47aef4d7d870d64caeb56bf65 P f99020fc71a943a6af10cb3407cb87b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99020fc71a943a6af10cb3407cb87b6" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 43609 } } peers { permanent_uuid: "aef17fb49def4c8d9ca13ac09d765375" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 41431 } } peers { permanent_uuid: "efc3a6eacf2e40f59b33a56e4ce46446" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42993 } }
I20260812 07:08:59.466738 29516 leader_election.cc:288] T 4454aeb47aef4d7d870d64caeb56bf65 P f99020fc71a943a6af10cb3407cb87b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aef17fb49def4c8d9ca13ac09d765375 (127.6.61.2:41431), efc3a6eacf2e40f59b33a56e4ce46446 (127.6.61.1:42993)
I20260812 07:08:59.473775 29509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:59.467326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:50077 (local address 127.6.61.1:42993)
0812 07:08:59.467591 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:59.467599 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:59.467993 (+ 394us) server_negotiation.cc:407] Connection header received
0812 07:08:59.468245 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:59.468254 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:59.468347 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:59.468475 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:08:59.468918 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:59.469771 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:59.471817 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:59.472700 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:59.472875 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:59.472991 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:59.473332 (+ 341us) server_negotiation.cc:299] Negotiation successful
0812 07:08:59.473479 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 07:08:59.474040 29494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:59.467335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:42993 (local address 127.6.61.3:50077)
0812 07:08:59.467837 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:59.467859 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:59.468047 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:59.468519 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:59.468525 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:59.468559 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:08:59.468756 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:59.468765 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:59.469958 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:59.469967 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:59.471630 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:59.471645 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:59.472893 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:59.472899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:59.473137 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:59.473165 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 07:08:59.473289 (+ 124us) client_negotiation.cc:240] Negotiation successful
0812 07:08:59.473454 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":38}
I20260812 07:08:59.474129 29507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:59.467358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:54459 (local address 127.6.61.2:41431)
0812 07:08:59.467605 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:59.467613 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:59.468378 (+ 765us) server_negotiation.cc:407] Connection header received
0812 07:08:59.468580 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:59.468584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:59.468644 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:59.468740 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:08:59.469275 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:59.470085 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:59.472024 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:59.472865 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:59.473033 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:59.473139 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:59.473599 (+ 460us) server_negotiation.cc:299] Negotiation successful
0812 07:08:59.473795 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 07:08:59.474291 29520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:59.467380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:41431 (local address 127.6.61.3:54459)
0812 07:08:59.468270 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0812 07:08:59.468285 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:08:59.468428 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:08:59.468773 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:08:59.468777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:08:59.468812 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:08:59.469111 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:59.469119 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:59.470250 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:59.470254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:59.471831 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:08:59.471845 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:08:59.473153 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:08:59.473163 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:08:59.473393 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:59.473423 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 07:08:59.473569 (+ 146us) client_negotiation.cc:240] Negotiation successful
0812 07:08:59.473722 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"mutex_wait_us":1,"thread_start_us":547,"threads_started":1}
I20260812 07:08:59.475400 29149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4454aeb47aef4d7d870d64caeb56bf65" candidate_uuid: "f99020fc71a943a6af10cb3407cb87b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efc3a6eacf2e40f59b33a56e4ce46446" is_pre_election: true
I20260812 07:08:59.475483 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4454aeb47aef4d7d870d64caeb56bf65" candidate_uuid: "f99020fc71a943a6af10cb3407cb87b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aef17fb49def4c8d9ca13ac09d765375" is_pre_election: true
I20260812 07:08:59.476009 29149 raft_consensus.cc:2454] T 4454aeb47aef4d7d870d64caeb56bf65 P efc3a6eacf2e40f59b33a56e4ce46446 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f99020fc71a943a6af10cb3407cb87b6 in term 0.
I20260812 07:08:59.476107 29292 raft_consensus.cc:2454] T 4454aeb47aef4d7d870d64caeb56bf65 P aef17fb49def4c8d9ca13ac09d765375 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f99020fc71a943a6af10cb3407cb87b6 in term 0.
I20260812 07:08:59.476861 29365 leader_election.cc:302] T 4454aeb47aef4d7d870d64caeb56bf65 P f99020fc71a943a6af10cb3407cb87b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aef17fb49def4c8d9ca13ac09d765375, f99020fc71a943a6af10cb3407cb87b6; no voters:
I20260812 07:08:59.477370 29516 raft_consensus.cc:2781] T 4454aeb47aef4d7d870d64caeb56bf65 P f99020fc71a943a6af10cb3407cb87b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:08:59.477551 29516 raft_consensus.cc:482] T 4454aeb47aef4d7d870d64caeb56bf65 P f99020fc71a943a6af10cb3407cb87b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:08:59.477689 29516 raft_consensus.cc:3037] T 4454aeb47aef4d7d870d64caeb56bf65 P f99020fc71a943a6af10cb3407cb87b6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:08:59.480248 29516 raft_consensus.cc:504] T 4454aeb47aef4d7d870d64caeb56bf65 P f99020fc71a943a6af10cb3407cb87b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99020fc71a943a6af10cb3407cb87b6" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 43609 } } peers { permanent_uuid: "aef17fb49def4c8d9ca13ac09d765375" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 41431 } } peers { permanent_uuid: "efc3a6eacf2e40f59b33a56e4ce46446" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42993 } }
I20260812 07:08:59.481062 29516 leader_election.cc:288] T 4454aeb47aef4d7d870d64caeb56bf65 P f99020fc71a943a6af10cb3407cb87b6 [CANDIDATE]: Term 1 election: Requested vote from peers aef17fb49def4c8d9ca13ac09d765375 (127.6.61.2:41431), efc3a6eacf2e40f59b33a56e4ce46446 (127.6.61.1:42993)
I20260812 07:08:59.481702 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4454aeb47aef4d7d870d64caeb56bf65" candidate_uuid: "f99020fc71a943a6af10cb3407cb87b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aef17fb49def4c8d9ca13ac09d765375"
I20260812 07:08:59.481912 29149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4454aeb47aef4d7d870d64caeb56bf65" candidate_uuid: "f99020fc71a943a6af10cb3407cb87b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efc3a6eacf2e40f59b33a56e4ce46446"
I20260812 07:08:59.481988 29292 raft_consensus.cc:3037] T 4454aeb47aef4d7d870d64caeb56bf65 P aef17fb49def4c8d9ca13ac09d765375 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:08:59.482193 29149 raft_consensus.cc:3037] T 4454aeb47aef4d7d870d64caeb56bf65 P efc3a6eacf2e40f59b33a56e4ce46446 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:08:59.484594 29292 raft_consensus.cc:2454] T 4454aeb47aef4d7d870d64caeb56bf65 P aef17fb49def4c8d9ca13ac09d765375 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f99020fc71a943a6af10cb3407cb87b6 in term 1.
I20260812 07:08:59.484825 29149 raft_consensus.cc:2454] T 4454aeb47aef4d7d870d64caeb56bf65 P efc3a6eacf2e40f59b33a56e4ce46446 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f99020fc71a943a6af10cb3407cb87b6 in term 1.
I20260812 07:08:59.485194 29365 leader_election.cc:302] T 4454aeb47aef4d7d870d64caeb56bf65 P f99020fc71a943a6af10cb3407cb87b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aef17fb49def4c8d9ca13ac09d765375, f99020fc71a943a6af10cb3407cb87b6; no voters:
I20260812 07:08:59.485602 29516 raft_consensus.cc:2781] T 4454aeb47aef4d7d870d64caeb56bf65 P f99020fc71a943a6af10cb3407cb87b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:08:59.486765 29516 raft_consensus.cc:686] T 4454aeb47aef4d7d870d64caeb56bf65 P f99020fc71a943a6af10cb3407cb87b6 [term 1 LEADER]: Becoming Leader. State: Replica: f99020fc71a943a6af10cb3407cb87b6, State: Running, Role: LEADER
I20260812 07:08:59.487808 29516 consensus_queue.cc:237] T 4454aeb47aef4d7d870d64caeb56bf65 P f99020fc71a943a6af10cb3407cb87b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99020fc71a943a6af10cb3407cb87b6" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 43609 } } peers { permanent_uuid: "aef17fb49def4c8d9ca13ac09d765375" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 41431 } } peers { permanent_uuid: "efc3a6eacf2e40f59b33a56e4ce46446" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42993 } }
I20260812 07:08:59.493127 28994 catalog_manager.cc:5495] T 4454aeb47aef4d7d870d64caeb56bf65 P f99020fc71a943a6af10cb3407cb87b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f99020fc71a943a6af10cb3407cb87b6 (127.6.61.3). New cstate: current_term: 1 leader_uuid: "f99020fc71a943a6af10cb3407cb87b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f99020fc71a943a6af10cb3407cb87b6" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 43609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aef17fb49def4c8d9ca13ac09d765375" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 41431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efc3a6eacf2e40f59b33a56e4ce46446" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42993 } health_report { overall_health: UNKNOWN } } }
I20260812 07:08:59.530442 29506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:59.524060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58072 (local address 127.6.61.3:43609)
0812 07:08:59.524296 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:59.524301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:59.524321 (+ 20us) server_negotiation.cc:407] Connection header received
0812 07:08:59.524540 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:59.524545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:59.524631 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:59.524788 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:08:59.525412 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:59.526384 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:59.527893 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:59.528429 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:59.528533 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:59.529017 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:08:59.529683 (+ 666us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:08:59.529803 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:59.530062 (+ 259us) server_negotiation.cc:299] Negotiation successful
0812 07:08:59.530135 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 07:08:59.539660 29292 raft_consensus.cc:1261] T 4454aeb47aef4d7d870d64caeb56bf65 P aef17fb49def4c8d9ca13ac09d765375 [term 1 FOLLOWER]: Refusing update from remote peer f99020fc71a943a6af10cb3407cb87b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:08:59.539660 29149 raft_consensus.cc:1261] T 4454aeb47aef4d7d870d64caeb56bf65 P efc3a6eacf2e40f59b33a56e4ce46446 [term 1 FOLLOWER]: Refusing update from remote peer f99020fc71a943a6af10cb3407cb87b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:08:59.540822 29521 consensus_queue.cc:1035] T 4454aeb47aef4d7d870d64caeb56bf65 P f99020fc71a943a6af10cb3407cb87b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aef17fb49def4c8d9ca13ac09d765375" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 41431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:08:59.541255 29516 consensus_queue.cc:1035] T 4454aeb47aef4d7d870d64caeb56bf65 P f99020fc71a943a6af10cb3407cb87b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efc3a6eacf2e40f59b33a56e4ce46446" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 42993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:08:59.554941 29525 mvcc.cc:205] Tried to move back new op lower bound from 7317579937942798336 to 7317579937746989056. Current Snapshot: MvccSnapshot[applied={T|T < 7317579937942798336}]
I20260812 07:08:59.562139 29527 mvcc.cc:205] Tried to move back new op lower bound from 7317579937942798336 to 7317579937746989056. Current Snapshot: MvccSnapshot[applied={T|T < 7317579937942798336}]
I20260812 07:08:59.572994 28994 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:08:59.573377 28994 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:08:59.578039 28994 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e76b6f5078f469a8ec4b503329b66eb: Sending DeleteTablet for 3 replicas of tablet 4454aeb47aef4d7d870d64caeb56bf65
I20260812 07:08:59.579550 29414 tablet_service.cc:1514] Processing DeleteTablet for tablet 4454aeb47aef4d7d870d64caeb56bf65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:08:59 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:41478
I20260812 07:08:59.579710 29128 tablet_service.cc:1514] Processing DeleteTablet for tablet 4454aeb47aef4d7d870d64caeb56bf65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:08:59 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:52336
I20260812 07:08:59.579722 29271 tablet_service.cc:1514] Processing DeleteTablet for tablet 4454aeb47aef4d7d870d64caeb56bf65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:08:59 UTC) from {username='kudu', principal='kudu/127.6.61.62@KRBTEST.COM'} at 127.0.0.1:60136
I20260812 07:08:59.585675 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 29058
I20260812 07:08:59.591228 29535 tablet_replica.cc:330] T 4454aeb47aef4d7d870d64caeb56bf65 P f99020fc71a943a6af10cb3407cb87b6: stopping tablet replica
I20260812 07:08:59.591564 29537 tablet_replica.cc:330] T 4454aeb47aef4d7d870d64caeb56bf65 P aef17fb49def4c8d9ca13ac09d765375: stopping tablet replica
I20260812 07:08:59.592089 29535 raft_consensus.cc:2229] T 4454aeb47aef4d7d870d64caeb56bf65 P f99020fc71a943a6af10cb3407cb87b6 [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:08:59.592386 29537 raft_consensus.cc:2229] T 4454aeb47aef4d7d870d64caeb56bf65 P aef17fb49def4c8d9ca13ac09d765375 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:08:59.593109 29535 raft_consensus.cc:2258] T 4454aeb47aef4d7d870d64caeb56bf65 P f99020fc71a943a6af10cb3407cb87b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:08:59.593169 29537 raft_consensus.cc:2258] T 4454aeb47aef4d7d870d64caeb56bf65 P aef17fb49def4c8d9ca13ac09d765375 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:08:59.594076 29536 tablet_replica.cc:330] T 4454aeb47aef4d7d870d64caeb56bf65 P efc3a6eacf2e40f59b33a56e4ce46446: stopping tablet replica
I20260812 07:08:59.594523 29536 raft_consensus.cc:2229] T 4454aeb47aef4d7d870d64caeb56bf65 P efc3a6eacf2e40f59b33a56e4ce46446 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:08:59.595088 29536 raft_consensus.cc:2258] T 4454aeb47aef4d7d870d64caeb56bf65 P efc3a6eacf2e40f59b33a56e4ce46446 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:08:59.595614 29537 ts_tablet_manager.cc:1905] T 4454aeb47aef4d7d870d64caeb56bf65 P aef17fb49def4c8d9ca13ac09d765375: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:08:59.597690 29536 ts_tablet_manager.cc:1905] T 4454aeb47aef4d7d870d64caeb56bf65 P efc3a6eacf2e40f59b33a56e4ce46446: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:08:59.599014 29535 ts_tablet_manager.cc:1905] T 4454aeb47aef4d7d870d64caeb56bf65 P f99020fc71a943a6af10cb3407cb87b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:08:59.605677 29536 ts_tablet_manager.cc:1918] T 4454aeb47aef4d7d870d64caeb56bf65 P efc3a6eacf2e40f59b33a56e4ce46446: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:08:59.606379 29536 log.cc:1196] T 4454aeb47aef4d7d870d64caeb56bf65 P efc3a6eacf2e40f59b33a56e4ce46446: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-0/wal/wals/4454aeb47aef4d7d870d64caeb56bf65
I20260812 07:08:59.607369 29536 ts_tablet_manager.cc:1939] T 4454aeb47aef4d7d870d64caeb56bf65 P efc3a6eacf2e40f59b33a56e4ce46446: Deleting consensus metadata
I20260812 07:08:59.609694 28977 catalog_manager.cc:4841] TS efc3a6eacf2e40f59b33a56e4ce46446 (127.6.61.1:42993): tablet 4454aeb47aef4d7d870d64caeb56bf65 (table test-table [id=9c967e25d4394329a376889e312c74f9]) successfully deleted
I20260812 07:08:59.610369 29535 ts_tablet_manager.cc:1918] T 4454aeb47aef4d7d870d64caeb56bf65 P f99020fc71a943a6af10cb3407cb87b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:08:59.610586 29535 log.cc:1196] T 4454aeb47aef4d7d870d64caeb56bf65 P f99020fc71a943a6af10cb3407cb87b6: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-2/wal/wals/4454aeb47aef4d7d870d64caeb56bf65
I20260812 07:08:59.611231 29535 ts_tablet_manager.cc:1939] T 4454aeb47aef4d7d870d64caeb56bf65 P f99020fc71a943a6af10cb3407cb87b6: Deleting consensus metadata
I20260812 07:08:59.612850 28978 catalog_manager.cc:4841] TS f99020fc71a943a6af10cb3407cb87b6 (127.6.61.3:43609): tablet 4454aeb47aef4d7d870d64caeb56bf65 (table test-table [id=9c967e25d4394329a376889e312c74f9]) successfully deleted
I20260812 07:08:59.613240 29537 ts_tablet_manager.cc:1918] T 4454aeb47aef4d7d870d64caeb56bf65 P aef17fb49def4c8d9ca13ac09d765375: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:08:59.613534 29537 log.cc:1196] T 4454aeb47aef4d7d870d64caeb56bf65 P aef17fb49def4c8d9ca13ac09d765375: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518208686283-6388-0/minicluster-data/ts-1/wal/wals/4454aeb47aef4d7d870d64caeb56bf65
Aug 12 07:08:59 dist-test-slave-69rg krb5kdc[28942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.1@KRBTEST.COM
I20260812 07:08:59.614257 29537 ts_tablet_manager.cc:1939] T 4454aeb47aef4d7d870d64caeb56bf65 P aef17fb49def4c8d9ca13ac09d765375: Deleting consensus metadata
I20260812 07:08:59.616340 28976 catalog_manager.cc:4841] TS aef17fb49def4c8d9ca13ac09d765375 (127.6.61.2:41431): tablet 4454aeb47aef4d7d870d64caeb56bf65 (table test-table [id=9c967e25d4394329a376889e312c74f9]) successfully deleted
I20260812 07:08:59.622249 29509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:59.587389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41360 (local address 127.6.61.1:42993)
0812 07:08:59.587660 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:59.587671 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:59.597499 (+ 9828us) server_negotiation.cc:407] Connection header received
0812 07:08:59.597942 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:59.597951 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:59.598059 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:59.598175 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:59.600304 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:59.601599 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:59.604044 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:59.604827 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:59.604939 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:59.615223 (+ 10284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:59.615238 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:59.615254 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:08:59.615294 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:59.618803 (+ 3509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:59.619677 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:59.619685 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:59.619688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:59.619836 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:59.620302 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:59.620307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:59.620311 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:59.620723 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:59.621099 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:59.621488 (+ 389us) server_negotiation.cc:299] Negotiation successful
0812 07:08:59.621767 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260812 07:08:59.623082 29193 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:08:59.719627 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 29058
I20260812 07:08:59.753494 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 29204
Aug 12 07:08:59 dist-test-slave-69rg krb5kdc[28942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.2@KRBTEST.COM
I20260812 07:08:59.775394 29507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:59.754508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45484 (local address 127.6.61.2:41431)
0812 07:08:59.754696 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:59.754702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:59.754728 (+ 26us) server_negotiation.cc:407] Connection header received
0812 07:08:59.755026 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:59.755030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:59.755117 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:59.755246 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:59.757316 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:59.758301 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:59.760671 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:59.761109 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:59.761253 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:59.768688 (+ 7435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:59.768710 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:59.768729 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:08:59.768778 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:59.772042 (+ 3264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:59.772942 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:59.772950 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:59.772953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:59.773098 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:59.773526 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:59.773531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:59.773536 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:59.774032 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:59.774357 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:59.774763 (+ 406us) server_negotiation.cc:299] Negotiation successful
0812 07:08:59.775015 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 07:08:59.776229 29334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:08:59.875119 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 29204
I20260812 07:08:59.907671 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 29347
Aug 12 07:08:59 dist-test-slave-69rg krb5kdc[28942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.3@KRBTEST.COM
I20260812 07:08:59.929472 29506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:08:59.908663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58076 (local address 127.6.61.3:43609)
0812 07:08:59.908859 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0812 07:08:59.908864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:08:59.908889 (+ 25us) server_negotiation.cc:407] Connection header received
0812 07:08:59.909253 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:08:59.909258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:08:59.909343 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:08:59.909437 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:08:59.911377 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:59.912329 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:59.914374 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:08:59.915060 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:08:59.915199 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:08:59.921520 (+ 6321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:08:59.921539 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:08:59.921559 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:08:59.921623 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:08:59.925668 (+ 4045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:59.926753 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:59.926762 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:59.926765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:59.927007 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:08:59.927634 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:08:59.927638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:08:59.927641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:08:59.928066 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:08:59.928401 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0812 07:08:59.928795 (+ 394us) server_negotiation.cc:299] Negotiation successful
0812 07:08:59.929074 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 07:08:59.930460 29484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:09:00.060346 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 29347
I20260812 07:09:00.092891 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 28957
I20260812 07:09:00.094288 29026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:09:00.211006 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 28957
2026-08-12T07:09:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (5146 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-12T07:09:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:09:00Z Disabled control of system clock
I20260812 07:09:00.310591 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:44701
--webserver_interface=127.6.61.62
--webserver_port=0
--builtin_ntp_servers=127.6.61.20:41019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:44701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 07:09:00.669315 29546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:09:00.669639 29546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:09:00.669860 29546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:09:00.677248 29546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:09:00.677358 29546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:09:00.677436 29546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:09:00.677553 29546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:09:00.677639 29546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:09:00.687072 29546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:41019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:44701
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:44701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:09:00.688493 29546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:09:00.690481 29546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:09:00.702883 29553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:00.702931 29554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:00.702961 29556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:09:00.704066 29546 server_base.cc:1034] running on GCE node
I20260812 07:09:00.705047 29546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:09:00.706837 29546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:09:00.708091 29546 hybrid_clock.cc:648] HybridClock initialized: now 1786518540708021 us; error 50 us; skew 500 ppm
I20260812 07:09:00.708570 29546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 07:09:00.711212 29546 webserver.cc:460] Webserver started at http://127.6.61.62:33463/ using document root <none> and password file <none>
I20260812 07:09:00.711974 29546 fs_manager.cc:359] Metadata directory not provided
I20260812 07:09:00.712070 29546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:09:00.712466 29546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:09:00.715559 29546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "8b6192ffb324496abdef4cd3d891e6ac"
format_stamp: "Formatted at 2026-08-12 07:09:00 on dist-test-slave-69rg"
server_key: "c67441f16fc15fd58f4cbe156257d921"
server_key_iv: "8f44f26753b988927a6b4d8d86b70482"
server_key_version: "encryptionkey@0"
I20260812 07:09:00.716513 29546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "8b6192ffb324496abdef4cd3d891e6ac"
format_stamp: "Formatted at 2026-08-12 07:09:00 on dist-test-slave-69rg"
server_key: "c67441f16fc15fd58f4cbe156257d921"
server_key_iv: "8f44f26753b988927a6b4d8d86b70482"
server_key_version: "encryptionkey@0"
I20260812 07:09:00.722159 29546 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260812 07:09:00.726323 29564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:09:00.727900 29546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 07:09:00.728134 29546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "8b6192ffb324496abdef4cd3d891e6ac"
format_stamp: "Formatted at 2026-08-12 07:09:00 on dist-test-slave-69rg"
server_key: "c67441f16fc15fd58f4cbe156257d921"
server_key_iv: "8f44f26753b988927a6b4d8d86b70482"
server_key_version: "encryptionkey@0"
I20260812 07:09:00.728384 29546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:09:00.747409 29546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:09:00.748348 29546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:09:00.748674 29546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:09:00.769585 29546 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:44701
I20260812 07:09:00.769625 29622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:44701 every 8 connection(s)
I20260812 07:09:00.771028 29546 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:09:00.775326 29623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:09:00.777372 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 29546
I20260812 07:09:00.777570 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:09:00.778272 6388 external_mini_cluster.cc:1428] Setting key ec5e6bdb45eb75ffa566943f487df30b
I20260812 07:09:00.788134 29623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b6192ffb324496abdef4cd3d891e6ac: Bootstrap starting.
I20260812 07:09:00.791466 29623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b6192ffb324496abdef4cd3d891e6ac: Neither blocks nor log segments found. Creating new log.
I20260812 07:09:00.792589 29623 log.cc:824] T 00000000000000000000000000000000 P 8b6192ffb324496abdef4cd3d891e6ac: Log is configured to *not* fsync() on all Append() calls
I20260812 07:09:00.792860 29626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:00.780405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50806 (local address 127.6.61.62:44701)
0812 07:09:00.781550 (+ 1145us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:00.781562 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:00.781617 (+ 55us) server_negotiation.cc:407] Connection header received
0812 07:09:00.783548 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:00.783573 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:00.784071 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:00.784482 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:00.785150 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:00.786619 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:00.788072 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:00.788884 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:00.789101 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:00.790385 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:00.790393 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:00.790408 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:09:00.790963 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:00.791057 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:09:00.791084 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:00.791224 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:00.791469 (+ 245us) server_negotiation.cc:299] Negotiation successful
0812 07:09:00.791847 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":414,"threads_started":1}
I20260812 07:09:00.795768 29623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b6192ffb324496abdef4cd3d891e6ac: No bootstrap required, opened a new log
I20260812 07:09:00.803648 29623 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b6192ffb324496abdef4cd3d891e6ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b6192ffb324496abdef4cd3d891e6ac" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 44701 } }
I20260812 07:09:00.804049 29623 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b6192ffb324496abdef4cd3d891e6ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:09:00.804119 29623 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b6192ffb324496abdef4cd3d891e6ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b6192ffb324496abdef4cd3d891e6ac, State: Initialized, Role: FOLLOWER
I20260812 07:09:00.804706 29623 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b6192ffb324496abdef4cd3d891e6ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b6192ffb324496abdef4cd3d891e6ac" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 44701 } }
I20260812 07:09:00.804927 29623 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b6192ffb324496abdef4cd3d891e6ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:09:00.805032 29623 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b6192ffb324496abdef4cd3d891e6ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:09:00.805177 29623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b6192ffb324496abdef4cd3d891e6ac [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:09:00.807509 29623 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b6192ffb324496abdef4cd3d891e6ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b6192ffb324496abdef4cd3d891e6ac" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 44701 } }
I20260812 07:09:00.808144 29623 leader_election.cc:302] T 00000000000000000000000000000000 P 8b6192ffb324496abdef4cd3d891e6ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b6192ffb324496abdef4cd3d891e6ac; no voters:
I20260812 07:09:00.808851 29623 leader_election.cc:288] T 00000000000000000000000000000000 P 8b6192ffb324496abdef4cd3d891e6ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:09:00.809041 29628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b6192ffb324496abdef4cd3d891e6ac [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:09:00.809957 29628 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b6192ffb324496abdef4cd3d891e6ac [term 1 LEADER]: Becoming Leader. State: Replica: 8b6192ffb324496abdef4cd3d891e6ac, State: Running, Role: LEADER
I20260812 07:09:00.810516 29628 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b6192ffb324496abdef4cd3d891e6ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b6192ffb324496abdef4cd3d891e6ac" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 44701 } }
I20260812 07:09:00.811113 29623 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b6192ffb324496abdef4cd3d891e6ac [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:09:00.814357 29630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b6192ffb324496abdef4cd3d891e6ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b6192ffb324496abdef4cd3d891e6ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b6192ffb324496abdef4cd3d891e6ac" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 44701 } } }
I20260812 07:09:00.814765 29630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b6192ffb324496abdef4cd3d891e6ac [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:00.815304 29631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b6192ffb324496abdef4cd3d891e6ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b6192ffb324496abdef4cd3d891e6ac. Latest consensus state: current_term: 1 leader_uuid: "8b6192ffb324496abdef4cd3d891e6ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b6192ffb324496abdef4cd3d891e6ac" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 44701 } } }
I20260812 07:09:00.815694 29631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b6192ffb324496abdef4cd3d891e6ac [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:00.825966 29637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:09:00.830534 29637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:09:00.841286 29637 catalog_manager.cc:1347] Generated new cluster ID: 0b5b728eb5fe47d78b0a1101e6fb2724
I20260812 07:09:00.841426 29637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:09:00.865973 29637 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:09:00.867394 29637 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:09:00.873525 29637 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b6192ffb324496abdef4cd3d891e6ac: Generated new TSK 0
I20260812 07:09:00.874785 29637 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:09:00.881282 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.1:0
--local_ip_for_outbound_sockets=127.6.61.1
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:44701
--builtin_ntp_servers=127.6.61.20:41019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 07:09:01.278872 29650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:09:01.279184 29650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:09:01.279340 29650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:09:01.287345 29650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:09:01.287478 29650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:09:01.287669 29650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.1
I20260812 07:09:01.298429 29650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:41019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.61.1
--webserver_port=0
--tserver_master_addrs=127.6.61.62:44701
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.1
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:09:01.299822 29650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:09:01.301769 29650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:09:01.317780 29655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:01.318090 29659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:01.318163 29657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:09:01.319005 29650 server_base.cc:1034] running on GCE node
I20260812 07:09:01.319671 29650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:09:01.320783 29650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:09:01.321987 29650 hybrid_clock.cc:648] HybridClock initialized: now 1786518541321917 us; error 60 us; skew 500 ppm
I20260812 07:09:01.322295 29650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 07:09:01.324842 29650 webserver.cc:460] Webserver started at http://127.6.61.1:41833/ using document root <none> and password file <none>
I20260812 07:09:01.325556 29650 fs_manager.cc:359] Metadata directory not provided
I20260812 07:09:01.325656 29650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:09:01.326032 29650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:09:01.329762 29650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-0/data/instance:
uuid: "4e2ddedf5c5e4c9aabbc1e34c5e32087"
format_stamp: "Formatted at 2026-08-12 07:09:01 on dist-test-slave-69rg"
server_key: "09d0ca024c69caba9166549e2461ee71"
server_key_iv: "11bf4d85bfdab995401a2dec55aa434a"
server_key_version: "encryptionkey@0"
I20260812 07:09:01.330837 29650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance:
uuid: "4e2ddedf5c5e4c9aabbc1e34c5e32087"
format_stamp: "Formatted at 2026-08-12 07:09:01 on dist-test-slave-69rg"
server_key: "09d0ca024c69caba9166549e2461ee71"
server_key_iv: "11bf4d85bfdab995401a2dec55aa434a"
server_key_version: "encryptionkey@0"
I20260812 07:09:01.336931 29650 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260812 07:09:01.340770 29665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:09:01.342397 29650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 07:09:01.342571 29650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-0/wal
uuid: "4e2ddedf5c5e4c9aabbc1e34c5e32087"
format_stamp: "Formatted at 2026-08-12 07:09:01 on dist-test-slave-69rg"
server_key: "09d0ca024c69caba9166549e2461ee71"
server_key_iv: "11bf4d85bfdab995401a2dec55aa434a"
server_key_version: "encryptionkey@0"
I20260812 07:09:01.342789 29650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:09:01.361876 29650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:09:01.362802 29650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:09:01.363122 29650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:09:01.364210 29650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:09:01.366000 29650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:09:01.366104 29650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:09:01.366362 29650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:09:01.366472 29650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:09:01.407539 29650 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.1:36975
I20260812 07:09:01.407588 29787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.1:36975 every 8 connection(s)
I20260812 07:09:01.409031 29650 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-0/data/info.pb
I20260812 07:09:01.410171 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 29650
I20260812 07:09:01.410356 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-0/wal/instance
I20260812 07:09:01.411000 6388 external_mini_cluster.cc:1428] Setting key 23fae0286643e090bb4c7eb40e4bc45b
I20260812 07:09:01.414721 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.2:0
--local_ip_for_outbound_sockets=127.6.61.2
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:44701
--builtin_ntp_servers=127.6.61.20:41019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 07:09:01.426195 29790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:01.412251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.1:47491 (local address 127.6.61.62:44701)
0812 07:09:01.412778 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:01.412782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:01.414619 (+ 1837us) server_negotiation.cc:407] Connection header received
0812 07:09:01.417554 (+ 2935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:01.417558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:01.417634 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:01.417757 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:01.419392 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:01.420213 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:01.422842 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:01.423268 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:01.423379 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:01.425195 (+ 1816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:01.425201 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:01.425205 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:09:01.425263 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:01.425312 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:09:01.425327 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:01.425412 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:01.425721 (+ 309us) server_negotiation.cc:299] Negotiation successful
0812 07:09:01.425899 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":273,"threads_started":1}
I20260812 07:09:01.427664 29792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:01.413085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:44701 (local address 127.6.61.1:47491)
0812 07:09:01.414438 (+ 1353us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:01.414490 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 07:09:01.417213 (+ 2723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:09:01.418058 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:09:01.418072 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:09:01.418243 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:09:01.419116 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:01.419144 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:01.420381 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:01.420389 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:01.422640 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:01.422660 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:01.423525 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:01.423535 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:01.423893 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:01.424726 (+ 833us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:09:01.424732 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:09:01.424791 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:09:01.424917 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:09:01.425442 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:09:01.425455 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:09:01.425467 (+ 12us) client_negotiation.cc:769] Sending connection context
0812 07:09:01.425662 (+ 195us) client_negotiation.cc:240] Negotiation successful
0812 07:09:01.426170 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":419,"threads_started":1}
I20260812 07:09:01.429647 29788 heartbeater.cc:347] Connected to a master server at 127.6.61.62:44701
I20260812 07:09:01.430007 29788 heartbeater.cc:464] Registering TS with master...
I20260812 07:09:01.430825 29788 heartbeater.cc:511] Master 127.6.61.62:44701 requested a full tablet report, sending...
I20260812 07:09:01.433338 29581 ts_manager.cc:194] Registered new tserver with Master: 4e2ddedf5c5e4c9aabbc1e34c5e32087 (127.6.61.1:36975)
I20260812 07:09:01.436022 29581 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.61.1:47491
W20260812 07:09:01.799542 29793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:09:01.799893 29793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:09:01.800004 29793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:09:01.810026 29793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:09:01.810189 29793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:09:01.810324 29793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.2
I20260812 07:09:01.821270 29793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:41019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.61.2
--webserver_port=0
--tserver_master_addrs=127.6.61.62:44701
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.2
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:09:01.822804 29793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:09:01.825237 29793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:09:01.842209 29799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:01.842135 29798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:01.842770 29801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:09:01.843251 29793 server_base.cc:1034] running on GCE node
I20260812 07:09:01.844506 29793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:09:01.845924 29793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:09:01.847249 29793 hybrid_clock.cc:648] HybridClock initialized: now 1786518541847184 us; error 54 us; skew 500 ppm
I20260812 07:09:01.847671 29793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 07:09:01.850667 29793 webserver.cc:460] Webserver started at http://127.6.61.2:33353/ using document root <none> and password file <none>
I20260812 07:09:01.851480 29793 fs_manager.cc:359] Metadata directory not provided
I20260812 07:09:01.851640 29793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:09:01.852007 29793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:09:01.855540 29793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-1/data/instance:
uuid: "e865901c556240bcbe8342880027e99b"
format_stamp: "Formatted at 2026-08-12 07:09:01 on dist-test-slave-69rg"
server_key: "01846e1840092821d5300b95a69a40cc"
server_key_iv: "d3e7a425f2d390d0fa99ed8a699af8d0"
server_key_version: "encryptionkey@0"
I20260812 07:09:01.856424 29793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance:
uuid: "e865901c556240bcbe8342880027e99b"
format_stamp: "Formatted at 2026-08-12 07:09:01 on dist-test-slave-69rg"
server_key: "01846e1840092821d5300b95a69a40cc"
server_key_iv: "d3e7a425f2d390d0fa99ed8a699af8d0"
server_key_version: "encryptionkey@0"
I20260812 07:09:01.862291 29793 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 07:09:01.867132 29809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:09:01.868733 29793 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 07:09:01.868983 29793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-1/wal
uuid: "e865901c556240bcbe8342880027e99b"
format_stamp: "Formatted at 2026-08-12 07:09:01 on dist-test-slave-69rg"
server_key: "01846e1840092821d5300b95a69a40cc"
server_key_iv: "d3e7a425f2d390d0fa99ed8a699af8d0"
server_key_version: "encryptionkey@0"
I20260812 07:09:01.869258 29793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:09:01.898831 29793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:09:01.899951 29793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:09:01.900245 29793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:09:01.901332 29793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:09:01.903033 29793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:09:01.903174 29793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:09:01.903314 29793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:09:01.903405 29793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:09:01.943089 29793 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.2:41381
I20260812 07:09:01.943140 29936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.2:41381 every 8 connection(s)
I20260812 07:09:01.944465 29793 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-1/data/info.pb
I20260812 07:09:01.947404 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 29793
I20260812 07:09:01.947623 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-1/wal/instance
I20260812 07:09:01.948331 6388 external_mini_cluster.cc:1428] Setting key 2bae44326a23020bff1a21bf8cb06ae6
I20260812 07:09:01.951871 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.61.3:0
--local_ip_for_outbound_sockets=127.6.61.3
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:44701
--builtin_ntp_servers=127.6.61.20:41019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 07:09:01.960740 29939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:01.947705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:47337 (local address 127.6.61.62:44701)
0812 07:09:01.948271 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:01.948278 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:01.949869 (+ 1591us) server_negotiation.cc:407] Connection header received
0812 07:09:01.952189 (+ 2320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:01.952196 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:01.952281 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:01.952352 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:01.953749 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:01.955054 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:01.956816 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:01.957632 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:01.957805 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:01.959595 (+ 1790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:01.959604 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:01.959609 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:09:01.959641 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:01.959706 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:09:01.959729 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:01.959838 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:01.960099 (+ 261us) server_negotiation.cc:299] Negotiation successful
0812 07:09:01.960245 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":262,"threads_started":1}
I20260812 07:09:01.961656 29940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:01.948377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:44701 (local address 127.6.61.2:47337)
0812 07:09:01.949601 (+ 1224us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:01.949663 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 07:09:01.951835 (+ 2172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:09:01.952805 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:09:01.952814 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:09:01.952909 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:09:01.953525 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:01.953547 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:01.955303 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:01.955307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:01.956632 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:01.956646 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:01.957927 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:01.957936 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:01.958276 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:01.959177 (+ 901us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:09:01.959182 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:09:01.959243 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:09:01.959350 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:09:01.959840 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:09:01.959852 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:09:01.959862 (+ 10us) client_negotiation.cc:769] Sending connection context
0812 07:09:01.960051 (+ 189us) client_negotiation.cc:240] Negotiation successful
0812 07:09:01.960497 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":360,"threads_started":1}
I20260812 07:09:01.963301 29937 heartbeater.cc:347] Connected to a master server at 127.6.61.62:44701
I20260812 07:09:01.963696 29937 heartbeater.cc:464] Registering TS with master...
I20260812 07:09:01.964766 29937 heartbeater.cc:511] Master 127.6.61.62:44701 requested a full tablet report, sending...
I20260812 07:09:01.966683 29581 ts_manager.cc:194] Registered new tserver with Master: e865901c556240bcbe8342880027e99b (127.6.61.2:41381)
I20260812 07:09:01.967586 29581 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.61.2:47337
W20260812 07:09:02.339962 29941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:09:02.340374 29941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:09:02.340549 29941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:09:02.348410 29941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:09:02.348563 29941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:09:02.348755 29941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.61.3
I20260812 07:09:02.358222 29941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:41019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.61.3
--webserver_port=0
--tserver_master_addrs=127.6.61.62:44701
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.61.3
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:09:02.359845 29941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:09:02.361861 29941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:09:02.378214 29946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:02.378125 29947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:02.378172 29949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:09:02.379222 29941 server_base.cc:1034] running on GCE node
I20260812 07:09:02.379958 29941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:09:02.381018 29941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:09:02.382246 29941 hybrid_clock.cc:648] HybridClock initialized: now 1786518542382185 us; error 40 us; skew 500 ppm
I20260812 07:09:02.382613 29941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 07:09:02.385375 29941 webserver.cc:460] Webserver started at http://127.6.61.3:39193/ using document root <none> and password file <none>
I20260812 07:09:02.386236 29941 fs_manager.cc:359] Metadata directory not provided
I20260812 07:09:02.386350 29941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:09:02.386747 29941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:09:02.389951 29941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-2/data/instance:
uuid: "5c4b1f0f844c4baaa4383c863e7d6042"
format_stamp: "Formatted at 2026-08-12 07:09:02 on dist-test-slave-69rg"
server_key: "dfbf6a6acdff4eb470d4f0e001e8a101"
server_key_iv: "6789c5d2e9a0782f87a3aa61b53811ab"
server_key_version: "encryptionkey@0"
I20260812 07:09:02.390852 29941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance:
uuid: "5c4b1f0f844c4baaa4383c863e7d6042"
format_stamp: "Formatted at 2026-08-12 07:09:02 on dist-test-slave-69rg"
server_key: "dfbf6a6acdff4eb470d4f0e001e8a101"
server_key_iv: "6789c5d2e9a0782f87a3aa61b53811ab"
server_key_version: "encryptionkey@0"
I20260812 07:09:02.398068 29941 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260812 07:09:02.403816 29955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:09:02.405650 29941 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260812 07:09:02.405923 29941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-2/wal
uuid: "5c4b1f0f844c4baaa4383c863e7d6042"
format_stamp: "Formatted at 2026-08-12 07:09:02 on dist-test-slave-69rg"
server_key: "dfbf6a6acdff4eb470d4f0e001e8a101"
server_key_iv: "6789c5d2e9a0782f87a3aa61b53811ab"
server_key_version: "encryptionkey@0"
I20260812 07:09:02.406219 29941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:09:02.434274 29941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:09:02.435333 29941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:09:02.435659 29941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:09:02.436779 29941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:09:02.438545 29941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:09:02.438679 29941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:09:02.438829 29941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:09:02.438925 29941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:09:02.439736 29788 heartbeater.cc:503] Master 127.6.61.62:44701 was elected leader, sending a full tablet report...
I20260812 07:09:02.477399 29941 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.3:44643
I20260812 07:09:02.477499 30078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.3:44643 every 8 connection(s)
I20260812 07:09:02.478847 29941 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-2/data/info.pb
I20260812 07:09:02.481103 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 29941
I20260812 07:09:02.481304 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-2/wal/instance
I20260812 07:09:02.481873 6388 external_mini_cluster.cc:1428] Setting key f5954040e7d5649e5afedaca2bc28b2b
I20260812 07:09:02.493198 30081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:02.482071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.3:57391 (local address 127.6.61.62:44701)
0812 07:09:02.482633 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:02.482640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:02.484252 (+ 1612us) server_negotiation.cc:407] Connection header received
0812 07:09:02.486262 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:02.486268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:02.486438 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:02.486558 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:02.487753 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:02.488556 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:02.490331 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:02.490793 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:02.490906 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:02.492371 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:02.492376 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:02.492379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:09:02.492401 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:02.492451 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:09:02.492463 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:02.492541 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:02.492808 (+ 267us) server_negotiation.cc:299] Negotiation successful
0812 07:09:02.492903 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":509,"threads_started":1}
I20260812 07:09:02.494331 30082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:02.482800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:44701 (local address 127.6.61.3:57391)
0812 07:09:02.484103 (+ 1303us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:02.484158 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 07:09:02.485964 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:09:02.486802 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:09:02.486811 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:09:02.486911 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:09:02.487550 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:02.487569 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:02.488717 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:02.488721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:02.490178 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:02.490193 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:02.491026 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:02.491032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:02.491242 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:02.491997 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:09:02.492000 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:09:02.492043 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:09:02.492146 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:09:02.492585 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:09:02.492595 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:09:02.492603 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 07:09:02.492793 (+ 190us) client_negotiation.cc:240] Negotiation successful
0812 07:09:02.493115 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"thread_start_us":484,"threads_started":1}
I20260812 07:09:02.495926 30079 heartbeater.cc:347] Connected to a master server at 127.6.61.62:44701
I20260812 07:09:02.496286 30079 heartbeater.cc:464] Registering TS with master...
I20260812 07:09:02.497225 30079 heartbeater.cc:511] Master 127.6.61.62:44701 requested a full tablet report, sending...
I20260812 07:09:02.499214 29581 ts_manager.cc:194] Registered new tserver with Master: 5c4b1f0f844c4baaa4383c863e7d6042 (127.6.61.3:44643)
I20260812 07:09:02.500128 29581 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.61.3:57391
I20260812 07:09:02.510673 6388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 07:09:02.521448 30081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:02.515309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50808 (local address 127.6.61.62:44701)
0812 07:09:02.515560 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:02.515568 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:02.515778 (+ 210us) server_negotiation.cc:407] Connection header received
0812 07:09:02.515927 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:02.515931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:02.516011 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:02.516095 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:02.516659 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:02.517386 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:02.519022 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:02.519490 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:02.519646 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:02.520639 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:02.520645 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:02.520649 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:09:02.520669 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:02.520701 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:09:02.520713 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:02.520787 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:02.520980 (+ 193us) server_negotiation.cc:299] Negotiation successful
0812 07:09:02.521116 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260812 07:09:02.526413 29581 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:09:02.533531 29581 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:09:02.578342 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:02.547750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.2:41381 (local address 127.0.0.1:52112)
0812 07:09:02.566590 (+ 18840us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:02.566627 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 07:09:02.566854 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:09:02.568778 (+ 1924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:09:02.568785 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:09:02.568818 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:09:02.569254 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:02.569270 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:02.572341 (+ 3071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:02.572348 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:02.575588 (+ 3240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:02.575608 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:02.577197 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:02.577203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:02.577416 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:02.577473 (+ 57us) client_negotiation.cc:769] Sending connection context
0812 07:09:02.577644 (+ 171us) client_negotiation.cc:240] Negotiation successful
0812 07:09:02.577906 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18629,"mutex_wait_us":23,"thread_start_us":20292,"threads_started":1}
I20260812 07:09:02.579258 30092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:02.547266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:36975 (local address 127.0.0.1:43966)
0812 07:09:02.548214 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:02.548282 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 07:09:02.548522 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:09:02.569244 (+ 20722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:09:02.569250 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:09:02.569282 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:09:02.569759 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:02.569778 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:02.571953 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:02.571959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:02.575401 (+ 3442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:02.575422 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:02.578273 (+ 2851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:02.578285 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:02.578514 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:02.578551 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 07:09:02.578643 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 07:09:02.578844 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":231,"thread_start_us":1060,"threads_started":1}
I20260812 07:09:02.579428 30094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:02.549352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:44643 (local address 127.0.0.1:55996)
0812 07:09:02.568396 (+ 19044us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:02.568427 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0812 07:09:02.568568 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:09:02.569543 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:09:02.569550 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:09:02.569580 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:09:02.570077 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:02.570090 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:02.572255 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:02.572260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:02.575718 (+ 3458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:02.575729 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:02.578487 (+ 2758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:02.578495 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:02.578718 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:02.578754 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 07:09:02.578843 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 07:09:02.578986 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18868,"mutex_wait_us":88,"thread_start_us":16802,"threads_started":1}
I20260812 07:09:02.579687 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:02.548055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52112 (local address 127.6.61.2:41381)
0812 07:09:02.567540 (+ 19485us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:02.567554 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:02.567719 (+ 165us) server_negotiation.cc:407] Connection header received
0812 07:09:02.567922 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:02.567935 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:02.568268 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:02.568533 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:09:02.569493 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:02.572145 (+ 2652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:02.575854 (+ 3709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:02.576961 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:02.577157 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:02.577446 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:02.577717 (+ 271us) server_negotiation.cc:299] Negotiation successful
0812 07:09:02.577921 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19227,"thread_start_us":19565,"threads_started":1}
I20260812 07:09:02.580228 30098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:02.549674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55996 (local address 127.6.61.3:44643)
0812 07:09:02.563210 (+ 13536us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:02.563219 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:02.568729 (+ 5510us) server_negotiation.cc:407] Connection header received
0812 07:09:02.568834 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:02.568841 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:02.569082 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:02.569298 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:09:02.570308 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:02.572067 (+ 1759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:02.576003 (+ 3936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:02.577731 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:02.577998 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:02.578343 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:02.579701 (+ 1358us) server_negotiation.cc:299] Negotiation successful
0812 07:09:02.579946 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13311,"thread_start_us":13116,"threads_started":1}
I20260812 07:09:02.581614 30093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:02.547416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43966 (local address 127.6.61.1:36975)
0812 07:09:02.568742 (+ 21326us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:02.568750 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:02.568783 (+ 33us) server_negotiation.cc:407] Connection header received
0812 07:09:02.568879 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:02.568884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:02.569030 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:02.569243 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:09:02.570118 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:02.571773 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:02.575683 (+ 3910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:02.578039 (+ 2356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:02.578210 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:02.578485 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:02.581042 (+ 2557us) server_negotiation.cc:299] Negotiation successful
0812 07:09:02.581233 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21146,"thread_start_us":21008,"threads_started":1}
I20260812 07:09:02.584376 30012 tablet_service.cc:1467] Processing CreateTablet for tablet 59622a6f5be64ad39e28a91065e44122 (DEFAULT_TABLE table=test-table [id=f34fe3d638534941ad24ed35a65bfd29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:09:02.584355 29865 tablet_service.cc:1467] Processing CreateTablet for tablet 59622a6f5be64ad39e28a91065e44122 (DEFAULT_TABLE table=test-table [id=f34fe3d638534941ad24ed35a65bfd29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:09:02.585940 29865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59622a6f5be64ad39e28a91065e44122. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:09:02.585954 30012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59622a6f5be64ad39e28a91065e44122. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:09:02.585533 29717 tablet_service.cc:1467] Processing CreateTablet for tablet 59622a6f5be64ad39e28a91065e44122 (DEFAULT_TABLE table=test-table [id=f34fe3d638534941ad24ed35a65bfd29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:09:02.587010 29717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59622a6f5be64ad39e28a91065e44122. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:09:02.599509 30099 tablet_bootstrap.cc:492] T 59622a6f5be64ad39e28a91065e44122 P 5c4b1f0f844c4baaa4383c863e7d6042: Bootstrap starting.
I20260812 07:09:02.603892 30101 tablet_bootstrap.cc:492] T 59622a6f5be64ad39e28a91065e44122 P e865901c556240bcbe8342880027e99b: Bootstrap starting.
I20260812 07:09:02.605022 30099 tablet_bootstrap.cc:654] T 59622a6f5be64ad39e28a91065e44122 P 5c4b1f0f844c4baaa4383c863e7d6042: Neither blocks nor log segments found. Creating new log.
I20260812 07:09:02.605929 30102 tablet_bootstrap.cc:492] T 59622a6f5be64ad39e28a91065e44122 P 4e2ddedf5c5e4c9aabbc1e34c5e32087: Bootstrap starting.
I20260812 07:09:02.606623 30099 log.cc:824] T 59622a6f5be64ad39e28a91065e44122 P 5c4b1f0f844c4baaa4383c863e7d6042: Log is configured to *not* fsync() on all Append() calls
I20260812 07:09:02.608180 30101 tablet_bootstrap.cc:654] T 59622a6f5be64ad39e28a91065e44122 P e865901c556240bcbe8342880027e99b: Neither blocks nor log segments found. Creating new log.
I20260812 07:09:02.609301 30102 tablet_bootstrap.cc:654] T 59622a6f5be64ad39e28a91065e44122 P 4e2ddedf5c5e4c9aabbc1e34c5e32087: Neither blocks nor log segments found. Creating new log.
I20260812 07:09:02.609683 30101 log.cc:824] T 59622a6f5be64ad39e28a91065e44122 P e865901c556240bcbe8342880027e99b: Log is configured to *not* fsync() on all Append() calls
I20260812 07:09:02.610664 30102 log.cc:824] T 59622a6f5be64ad39e28a91065e44122 P 4e2ddedf5c5e4c9aabbc1e34c5e32087: Log is configured to *not* fsync() on all Append() calls
I20260812 07:09:02.610764 30099 tablet_bootstrap.cc:492] T 59622a6f5be64ad39e28a91065e44122 P 5c4b1f0f844c4baaa4383c863e7d6042: No bootstrap required, opened a new log
I20260812 07:09:02.611107 30099 ts_tablet_manager.cc:1397] T 59622a6f5be64ad39e28a91065e44122 P 5c4b1f0f844c4baaa4383c863e7d6042: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260812 07:09:02.614063 30101 tablet_bootstrap.cc:492] T 59622a6f5be64ad39e28a91065e44122 P e865901c556240bcbe8342880027e99b: No bootstrap required, opened a new log
I20260812 07:09:02.614504 30101 ts_tablet_manager.cc:1397] T 59622a6f5be64ad39e28a91065e44122 P e865901c556240bcbe8342880027e99b: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260812 07:09:02.621325 30102 tablet_bootstrap.cc:492] T 59622a6f5be64ad39e28a91065e44122 P 4e2ddedf5c5e4c9aabbc1e34c5e32087: No bootstrap required, opened a new log
I20260812 07:09:02.621661 30102 ts_tablet_manager.cc:1397] T 59622a6f5be64ad39e28a91065e44122 P 4e2ddedf5c5e4c9aabbc1e34c5e32087: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.004s
I20260812 07:09:02.624686 30099 raft_consensus.cc:348] T 59622a6f5be64ad39e28a91065e44122 P 5c4b1f0f844c4baaa4383c863e7d6042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2ddedf5c5e4c9aabbc1e34c5e32087" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 36975 } } peers { permanent_uuid: "5c4b1f0f844c4baaa4383c863e7d6042" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 44643 } } peers { permanent_uuid: "e865901c556240bcbe8342880027e99b" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 41381 } }
I20260812 07:09:02.625262 30099 raft_consensus.cc:374] T 59622a6f5be64ad39e28a91065e44122 P 5c4b1f0f844c4baaa4383c863e7d6042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:09:02.625434 30099 raft_consensus.cc:729] T 59622a6f5be64ad39e28a91065e44122 P 5c4b1f0f844c4baaa4383c863e7d6042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c4b1f0f844c4baaa4383c863e7d6042, State: Initialized, Role: FOLLOWER
I20260812 07:09:02.626346 30099 consensus_queue.cc:260] T 59622a6f5be64ad39e28a91065e44122 P 5c4b1f0f844c4baaa4383c863e7d6042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2ddedf5c5e4c9aabbc1e34c5e32087" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 36975 } } peers { permanent_uuid: "5c4b1f0f844c4baaa4383c863e7d6042" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 44643 } } peers { permanent_uuid: "e865901c556240bcbe8342880027e99b" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 41381 } }
I20260812 07:09:02.626585 30101 raft_consensus.cc:348] T 59622a6f5be64ad39e28a91065e44122 P e865901c556240bcbe8342880027e99b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2ddedf5c5e4c9aabbc1e34c5e32087" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 36975 } } peers { permanent_uuid: "5c4b1f0f844c4baaa4383c863e7d6042" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 44643 } } peers { permanent_uuid: "e865901c556240bcbe8342880027e99b" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 41381 } }
I20260812 07:09:02.627233 30101 raft_consensus.cc:374] T 59622a6f5be64ad39e28a91065e44122 P e865901c556240bcbe8342880027e99b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:09:02.627413 30101 raft_consensus.cc:729] T 59622a6f5be64ad39e28a91065e44122 P e865901c556240bcbe8342880027e99b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e865901c556240bcbe8342880027e99b, State: Initialized, Role: FOLLOWER
I20260812 07:09:02.627894 30079 heartbeater.cc:503] Master 127.6.61.62:44701 was elected leader, sending a full tablet report...
I20260812 07:09:02.628207 30101 consensus_queue.cc:260] T 59622a6f5be64ad39e28a91065e44122 P e865901c556240bcbe8342880027e99b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2ddedf5c5e4c9aabbc1e34c5e32087" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 36975 } } peers { permanent_uuid: "5c4b1f0f844c4baaa4383c863e7d6042" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 44643 } } peers { permanent_uuid: "e865901c556240bcbe8342880027e99b" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 41381 } }
I20260812 07:09:02.628772 30099 ts_tablet_manager.cc:1428] T 59622a6f5be64ad39e28a91065e44122 P 5c4b1f0f844c4baaa4383c863e7d6042: Time spent starting tablet: real 0.018s user 0.010s sys 0.005s
I20260812 07:09:02.629942 29937 heartbeater.cc:503] Master 127.6.61.62:44701 was elected leader, sending a full tablet report...
I20260812 07:09:02.631270 30101 ts_tablet_manager.cc:1428] T 59622a6f5be64ad39e28a91065e44122 P e865901c556240bcbe8342880027e99b: Time spent starting tablet: real 0.017s user 0.011s sys 0.004s
I20260812 07:09:02.633476 30107 raft_consensus.cc:482] T 59622a6f5be64ad39e28a91065e44122 P e865901c556240bcbe8342880027e99b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:09:02.633881 30107 raft_consensus.cc:504] T 59622a6f5be64ad39e28a91065e44122 P e865901c556240bcbe8342880027e99b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2ddedf5c5e4c9aabbc1e34c5e32087" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 36975 } } peers { permanent_uuid: "5c4b1f0f844c4baaa4383c863e7d6042" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 44643 } } peers { permanent_uuid: "e865901c556240bcbe8342880027e99b" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 41381 } }
I20260812 07:09:02.635340 30102 raft_consensus.cc:348] T 59622a6f5be64ad39e28a91065e44122 P 4e2ddedf5c5e4c9aabbc1e34c5e32087 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2ddedf5c5e4c9aabbc1e34c5e32087" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 36975 } } peers { permanent_uuid: "5c4b1f0f844c4baaa4383c863e7d6042" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 44643 } } peers { permanent_uuid: "e865901c556240bcbe8342880027e99b" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 41381 } }
I20260812 07:09:02.635936 30102 raft_consensus.cc:374] T 59622a6f5be64ad39e28a91065e44122 P 4e2ddedf5c5e4c9aabbc1e34c5e32087 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:09:02.636056 30102 raft_consensus.cc:729] T 59622a6f5be64ad39e28a91065e44122 P 4e2ddedf5c5e4c9aabbc1e34c5e32087 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e2ddedf5c5e4c9aabbc1e34c5e32087, State: Initialized, Role: FOLLOWER
I20260812 07:09:02.636901 30102 consensus_queue.cc:260] T 59622a6f5be64ad39e28a91065e44122 P 4e2ddedf5c5e4c9aabbc1e34c5e32087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2ddedf5c5e4c9aabbc1e34c5e32087" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 36975 } } peers { permanent_uuid: "5c4b1f0f844c4baaa4383c863e7d6042" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 44643 } } peers { permanent_uuid: "e865901c556240bcbe8342880027e99b" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 41381 } }
I20260812 07:09:02.640434 30102 ts_tablet_manager.cc:1428] T 59622a6f5be64ad39e28a91065e44122 P 4e2ddedf5c5e4c9aabbc1e34c5e32087: Time spent starting tablet: real 0.019s user 0.009s sys 0.008s
I20260812 07:09:02.641084 30107 leader_election.cc:288] T 59622a6f5be64ad39e28a91065e44122 P e865901c556240bcbe8342880027e99b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e2ddedf5c5e4c9aabbc1e34c5e32087 (127.6.61.1:36975), 5c4b1f0f844c4baaa4383c863e7d6042 (127.6.61.3:44643)
I20260812 07:09:02.648386 30093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:02.637065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:35569 (local address 127.6.61.1:36975)
0812 07:09:02.637287 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:02.637294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:02.640043 (+ 2749us) server_negotiation.cc:407] Connection header received
0812 07:09:02.640371 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:02.640377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:02.640490 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:02.640564 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:09:02.641493 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:02.642891 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:02.645887 (+ 2996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:02.647032 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:02.647190 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:02.647302 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:02.648028 (+ 726us) server_negotiation.cc:299] Negotiation successful
0812 07:09:02.648178 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 07:09:02.648557 30108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:02.636537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.1:36975 (local address 127.6.61.2:35569)
0812 07:09:02.639939 (+ 3402us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:02.639961 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 07:09:02.640169 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:09:02.640765 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:09:02.640773 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:09:02.640806 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:09:02.641288 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:02.641304 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:02.643283 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:02.643290 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:02.645563 (+ 2273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:02.645581 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:02.647581 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:02.647596 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:02.647821 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:02.647859 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 07:09:02.647955 (+ 96us) client_negotiation.cc:240] Negotiation successful
0812 07:09:02.648141 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3224,"mutex_wait_us":43,"thread_start_us":3059,"threads_started":1}
I20260812 07:09:02.648952 30098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:02.640541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.61.2:32873 (local address 127.6.61.3:44643)
0812 07:09:02.640769 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:02.640775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:02.641923 (+ 1148us) server_negotiation.cc:407] Connection header received
0812 07:09:02.642124 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:02.642128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:02.642191 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:02.642269 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:09:02.643498 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:02.644210 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:02.646715 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:02.647886 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:02.648042 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:02.648143 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:02.648567 (+ 424us) server_negotiation.cc:299] Negotiation successful
0812 07:09:02.648728 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 07:09:02.648972 30110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:02.640918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.3:44643 (local address 127.6.61.2:32873)
0812 07:09:02.641799 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:02.641847 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 07:09:02.641971 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:09:02.642507 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:09:02.642517 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:09:02.642573 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:09:02.643295 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:02.643311 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:02.644374 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:02.644381 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:02.646511 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:02.646524 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:02.648112 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:02.648119 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:02.648342 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:02.648375 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 07:09:02.648481 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 07:09:02.648634 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"mutex_wait_us":1,"thread_start_us":558,"threads_started":1}
I20260812 07:09:02.649986 29739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59622a6f5be64ad39e28a91065e44122" candidate_uuid: "e865901c556240bcbe8342880027e99b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e2ddedf5c5e4c9aabbc1e34c5e32087" is_pre_election: true
I20260812 07:09:02.650146 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59622a6f5be64ad39e28a91065e44122" candidate_uuid: "e865901c556240bcbe8342880027e99b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c4b1f0f844c4baaa4383c863e7d6042" is_pre_election: true
I20260812 07:09:02.650673 29739 raft_consensus.cc:2454] T 59622a6f5be64ad39e28a91065e44122 P 4e2ddedf5c5e4c9aabbc1e34c5e32087 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e865901c556240bcbe8342880027e99b in term 0.
I20260812 07:09:02.650691 30032 raft_consensus.cc:2454] T 59622a6f5be64ad39e28a91065e44122 P 5c4b1f0f844c4baaa4383c863e7d6042 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e865901c556240bcbe8342880027e99b in term 0.
I20260812 07:09:02.651401 29810 leader_election.cc:302] T 59622a6f5be64ad39e28a91065e44122 P e865901c556240bcbe8342880027e99b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c4b1f0f844c4baaa4383c863e7d6042, e865901c556240bcbe8342880027e99b; no voters:
I20260812 07:09:02.651809 30107 raft_consensus.cc:2781] T 59622a6f5be64ad39e28a91065e44122 P e865901c556240bcbe8342880027e99b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:09:02.651973 30107 raft_consensus.cc:482] T 59622a6f5be64ad39e28a91065e44122 P e865901c556240bcbe8342880027e99b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:09:02.652094 30107 raft_consensus.cc:3037] T 59622a6f5be64ad39e28a91065e44122 P e865901c556240bcbe8342880027e99b [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:09:02.655023 30107 raft_consensus.cc:504] T 59622a6f5be64ad39e28a91065e44122 P e865901c556240bcbe8342880027e99b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2ddedf5c5e4c9aabbc1e34c5e32087" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 36975 } } peers { permanent_uuid: "5c4b1f0f844c4baaa4383c863e7d6042" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 44643 } } peers { permanent_uuid: "e865901c556240bcbe8342880027e99b" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 41381 } }
I20260812 07:09:02.655743 30107 leader_election.cc:288] T 59622a6f5be64ad39e28a91065e44122 P e865901c556240bcbe8342880027e99b [CANDIDATE]: Term 1 election: Requested vote from peers 4e2ddedf5c5e4c9aabbc1e34c5e32087 (127.6.61.1:36975), 5c4b1f0f844c4baaa4383c863e7d6042 (127.6.61.3:44643)
I20260812 07:09:02.656243 29739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59622a6f5be64ad39e28a91065e44122" candidate_uuid: "e865901c556240bcbe8342880027e99b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e2ddedf5c5e4c9aabbc1e34c5e32087"
I20260812 07:09:02.656399 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59622a6f5be64ad39e28a91065e44122" candidate_uuid: "e865901c556240bcbe8342880027e99b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c4b1f0f844c4baaa4383c863e7d6042"
I20260812 07:09:02.656455 29739 raft_consensus.cc:3037] T 59622a6f5be64ad39e28a91065e44122 P 4e2ddedf5c5e4c9aabbc1e34c5e32087 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:09:02.656646 30032 raft_consensus.cc:3037] T 59622a6f5be64ad39e28a91065e44122 P 5c4b1f0f844c4baaa4383c863e7d6042 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:09:02.658939 29739 raft_consensus.cc:2454] T 59622a6f5be64ad39e28a91065e44122 P 4e2ddedf5c5e4c9aabbc1e34c5e32087 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e865901c556240bcbe8342880027e99b in term 1.
I20260812 07:09:02.659283 30032 raft_consensus.cc:2454] T 59622a6f5be64ad39e28a91065e44122 P 5c4b1f0f844c4baaa4383c863e7d6042 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e865901c556240bcbe8342880027e99b in term 1.
I20260812 07:09:02.659703 29810 leader_election.cc:302] T 59622a6f5be64ad39e28a91065e44122 P e865901c556240bcbe8342880027e99b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e2ddedf5c5e4c9aabbc1e34c5e32087, e865901c556240bcbe8342880027e99b; no voters:
I20260812 07:09:02.660125 30107 raft_consensus.cc:2781] T 59622a6f5be64ad39e28a91065e44122 P e865901c556240bcbe8342880027e99b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:09:02.660677 30107 raft_consensus.cc:686] T 59622a6f5be64ad39e28a91065e44122 P e865901c556240bcbe8342880027e99b [term 1 LEADER]: Becoming Leader. State: Replica: e865901c556240bcbe8342880027e99b, State: Running, Role: LEADER
I20260812 07:09:02.661247 30107 consensus_queue.cc:237] T 59622a6f5be64ad39e28a91065e44122 P e865901c556240bcbe8342880027e99b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2ddedf5c5e4c9aabbc1e34c5e32087" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 36975 } } peers { permanent_uuid: "5c4b1f0f844c4baaa4383c863e7d6042" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 44643 } } peers { permanent_uuid: "e865901c556240bcbe8342880027e99b" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 41381 } }
W20260812 07:09:02.662292 29789 tablet.cc:2369] T 59622a6f5be64ad39e28a91065e44122 P 4e2ddedf5c5e4c9aabbc1e34c5e32087: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:09:02.666132 29581 catalog_manager.cc:5495] T 59622a6f5be64ad39e28a91065e44122 P e865901c556240bcbe8342880027e99b reported cstate change: term changed from 0 to 1, leader changed from <none> to e865901c556240bcbe8342880027e99b (127.6.61.2). New cstate: current_term: 1 leader_uuid: "e865901c556240bcbe8342880027e99b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2ddedf5c5e4c9aabbc1e34c5e32087" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 36975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c4b1f0f844c4baaa4383c863e7d6042" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 44643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e865901c556240bcbe8342880027e99b" member_type: VOTER last_known_addr { host: "127.6.61.2" port: 41381 } health_report { overall_health: HEALTHY } } }
I20260812 07:09:02.689977 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:02.684064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52116 (local address 127.6.61.2:41381)
0812 07:09:02.684274 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:02.684279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:02.684299 (+ 20us) server_negotiation.cc:407] Connection header received
0812 07:09:02.684541 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:02.684546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:02.684685 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:02.684846 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:09:02.685189 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:02.686158 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:02.687600 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:02.688021 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:02.688178 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:02.688679 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:09:02.689260 (+ 581us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:09:02.689395 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:02.689547 (+ 152us) server_negotiation.cc:299] Negotiation successful
0812 07:09:02.689623 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260812 07:09:02.697265 29938 tablet.cc:2369] T 59622a6f5be64ad39e28a91065e44122 P e865901c556240bcbe8342880027e99b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:09:02.699066 29739 raft_consensus.cc:1261] T 59622a6f5be64ad39e28a91065e44122 P 4e2ddedf5c5e4c9aabbc1e34c5e32087 [term 1 FOLLOWER]: Refusing update from remote peer e865901c556240bcbe8342880027e99b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:09:02.699505 30032 raft_consensus.cc:1261] T 59622a6f5be64ad39e28a91065e44122 P 5c4b1f0f844c4baaa4383c863e7d6042 [term 1 FOLLOWER]: Refusing update from remote peer e865901c556240bcbe8342880027e99b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:09:02.700363 30107 consensus_queue.cc:1035] T 59622a6f5be64ad39e28a91065e44122 P e865901c556240bcbe8342880027e99b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e2ddedf5c5e4c9aabbc1e34c5e32087" member_type: VOTER last_known_addr { host: "127.6.61.1" port: 36975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:09:02.700708 30111 consensus_queue.cc:1035] T 59622a6f5be64ad39e28a91065e44122 P e865901c556240bcbe8342880027e99b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c4b1f0f844c4baaa4383c863e7d6042" member_type: VOTER last_known_addr { host: "127.6.61.3" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:09:02.715076 30114 mvcc.cc:205] Tried to move back new op lower bound from 7317579950883311616 to 7317579950745411584. Current Snapshot: MvccSnapshot[applied={T|T < 7317579950883311616}]
I20260812 07:09:02.717034 30116 mvcc.cc:205] Tried to move back new op lower bound from 7317579950883311616 to 7317579950745411584. Current Snapshot: MvccSnapshot[applied={T|T < 7317579950883311616}]
I20260812 07:09:02.732475 29581 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:09:02.732782 29581 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:09:02.736701 29581 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b6192ffb324496abdef4cd3d891e6ac: Sending DeleteTablet for 3 replicas of tablet 59622a6f5be64ad39e28a91065e44122
I20260812 07:09:02.738190 30012 tablet_service.cc:1514] Processing DeleteTablet for tablet 59622a6f5be64ad39e28a91065e44122 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:09:02 UTC) from {username='slave'} at 127.0.0.1:55996
I20260812 07:09:02.738207 29865 tablet_service.cc:1514] Processing DeleteTablet for tablet 59622a6f5be64ad39e28a91065e44122 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:09:02 UTC) from {username='slave'} at 127.0.0.1:52112
I20260812 07:09:02.738252 29717 tablet_service.cc:1514] Processing DeleteTablet for tablet 59622a6f5be64ad39e28a91065e44122 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:09:02 UTC) from {username='slave'} at 127.0.0.1:43966
I20260812 07:09:02.740607 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 29650
I20260812 07:09:02.751169 30129 tablet_replica.cc:330] T 59622a6f5be64ad39e28a91065e44122 P 5c4b1f0f844c4baaa4383c863e7d6042: stopping tablet replica
I20260812 07:09:02.751915 30129 raft_consensus.cc:2229] T 59622a6f5be64ad39e28a91065e44122 P 5c4b1f0f844c4baaa4383c863e7d6042 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:09:02.752712 30129 raft_consensus.cc:2258] T 59622a6f5be64ad39e28a91065e44122 P 5c4b1f0f844c4baaa4383c863e7d6042 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:09:02.753834 30131 tablet_replica.cc:330] T 59622a6f5be64ad39e28a91065e44122 P 4e2ddedf5c5e4c9aabbc1e34c5e32087: stopping tablet replica
I20260812 07:09:02.754479 30131 raft_consensus.cc:2229] T 59622a6f5be64ad39e28a91065e44122 P 4e2ddedf5c5e4c9aabbc1e34c5e32087 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:09:02.755187 30131 raft_consensus.cc:2258] T 59622a6f5be64ad39e28a91065e44122 P 4e2ddedf5c5e4c9aabbc1e34c5e32087 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:09:02.758198 30131 ts_tablet_manager.cc:1905] T 59622a6f5be64ad39e28a91065e44122 P 4e2ddedf5c5e4c9aabbc1e34c5e32087: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:09:02.759127 30130 tablet_replica.cc:330] T 59622a6f5be64ad39e28a91065e44122 P e865901c556240bcbe8342880027e99b: stopping tablet replica
I20260812 07:09:02.759723 30130 raft_consensus.cc:2229] T 59622a6f5be64ad39e28a91065e44122 P e865901c556240bcbe8342880027e99b [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:09:02.760562 30130 raft_consensus.cc:2258] T 59622a6f5be64ad39e28a91065e44122 P e865901c556240bcbe8342880027e99b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:09:02.760596 30129 ts_tablet_manager.cc:1905] T 59622a6f5be64ad39e28a91065e44122 P 5c4b1f0f844c4baaa4383c863e7d6042: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:09:02.762847 30130 ts_tablet_manager.cc:1905] T 59622a6f5be64ad39e28a91065e44122 P e865901c556240bcbe8342880027e99b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:09:02.773217 30131 ts_tablet_manager.cc:1918] T 59622a6f5be64ad39e28a91065e44122 P 4e2ddedf5c5e4c9aabbc1e34c5e32087: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:09:02.773837 30129 ts_tablet_manager.cc:1918] T 59622a6f5be64ad39e28a91065e44122 P 5c4b1f0f844c4baaa4383c863e7d6042: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:09:02.774228 30129 log.cc:1196] T 59622a6f5be64ad39e28a91065e44122 P 5c4b1f0f844c4baaa4383c863e7d6042: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-2/wal/wals/59622a6f5be64ad39e28a91065e44122
I20260812 07:09:02.774945 30129 ts_tablet_manager.cc:1939] T 59622a6f5be64ad39e28a91065e44122 P 5c4b1f0f844c4baaa4383c863e7d6042: Deleting consensus metadata
I20260812 07:09:02.776188 30093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:02.742055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43972 (local address 127.6.61.1:36975)
0812 07:09:02.742246 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:02.742253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:02.758905 (+ 16652us) server_negotiation.cc:407] Connection header received
0812 07:09:02.759468 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:02.759475 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:02.759589 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:02.759676 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:02.760426 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:02.761570 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:02.763652 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:02.764357 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:02.764467 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:02.774136 (+ 9669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:02.774151 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:02.774174 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:09:02.775040 (+ 866us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:02.775123 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:09:02.775153 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:02.775258 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:02.775630 (+ 372us) server_negotiation.cc:299] Negotiation successful
0812 07:09:02.775793 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 07:09:02.777380 30131 log.cc:1196] T 59622a6f5be64ad39e28a91065e44122 P 4e2ddedf5c5e4c9aabbc1e34c5e32087: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-0/wal/wals/59622a6f5be64ad39e28a91065e44122
I20260812 07:09:02.777503 29565 catalog_manager.cc:4841] TS 5c4b1f0f844c4baaa4383c863e7d6042 (127.6.61.3:44643): tablet 59622a6f5be64ad39e28a91065e44122 (table test-table [id=f34fe3d638534941ad24ed35a65bfd29]) successfully deleted
I20260812 07:09:02.778173 30131 ts_tablet_manager.cc:1939] T 59622a6f5be64ad39e28a91065e44122 P 4e2ddedf5c5e4c9aabbc1e34c5e32087: Deleting consensus metadata
I20260812 07:09:02.778208 29783 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:09:02.781842 30130 ts_tablet_manager.cc:1918] T 59622a6f5be64ad39e28a91065e44122 P e865901c556240bcbe8342880027e99b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:09:02.782234 30130 log.cc:1196] T 59622a6f5be64ad39e28a91065e44122 P e865901c556240bcbe8342880027e99b: Deleting WAL directory at /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518208686283-6388-0/minicluster-data/ts-1/wal/wals/59622a6f5be64ad39e28a91065e44122
I20260812 07:09:02.783196 30130 ts_tablet_manager.cc:1939] T 59622a6f5be64ad39e28a91065e44122 P e865901c556240bcbe8342880027e99b: Deleting consensus metadata
I20260812 07:09:02.786032 29568 catalog_manager.cc:4841] TS e865901c556240bcbe8342880027e99b (127.6.61.2:41381): tablet 59622a6f5be64ad39e28a91065e44122 (table test-table [id=f34fe3d638534941ad24ed35a65bfd29]) successfully deleted
I20260812 07:09:02.885300 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 29650
I20260812 07:09:02.886813 29565 catalog_manager.cc:4841] TS 4e2ddedf5c5e4c9aabbc1e34c5e32087 (127.6.61.1:36975): tablet 59622a6f5be64ad39e28a91065e44122 (table test-table [id=f34fe3d638534941ad24ed35a65bfd29]) successfully deleted
I20260812 07:09:02.913807 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 29793
I20260812 07:09:02.922648 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:02.914744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52130 (local address 127.6.61.2:41381)
0812 07:09:02.914964 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:02.914971 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:02.914995 (+ 24us) server_negotiation.cc:407] Connection header received
0812 07:09:02.915160 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:02.915167 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:02.915275 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:02.915398 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:02.915867 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:02.917024 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:02.918749 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:02.919431 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:02.919602 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:02.920809 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:02.920820 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:02.920841 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:09:02.921681 (+ 840us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:02.921746 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:09:02.921772 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:02.921940 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:02.922093 (+ 153us) server_negotiation.cc:299] Negotiation successful
0812 07:09:02.922204 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 07:09:02.923485 29932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:09:03.043695 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 29793
I20260812 07:09:03.068429 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 29941
I20260812 07:09:03.077167 30098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:03.069377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55998 (local address 127.6.61.3:44643)
0812 07:09:03.069746 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:03.069753 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:03.069786 (+ 33us) server_negotiation.cc:407] Connection header received
0812 07:09:03.069973 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:03.069979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:03.070114 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:03.070267 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:03.071055 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:03.071932 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:03.073538 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:03.074192 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:03.074312 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:03.075558 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:03.075569 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:03.075586 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:09:03.076204 (+ 618us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:03.076306 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:09:03.076329 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:03.076425 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:03.076622 (+ 197us) server_negotiation.cc:299] Negotiation successful
0812 07:09:03.076745 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260812 07:09:03.078106 30074 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:09:03.170966 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 29941
I20260812 07:09:03.195334 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 29546
I20260812 07:09:03.196615 29617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:09:03.307801 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 29546
2026-08-12T07:09:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3097 ms)
[----------] 6 tests from EncryptionPolicyTest (24372 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:09:03 dist-test-slave-69rg krb5kdc[30139](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:09:03 dist-test-slave-69rg krb5kdc[30139](info): set up 2 sockets
Aug 12 07:09:03 dist-test-slave-69rg krb5kdc[30139](info): commencing operation
krb5kdc: starting...
W20260812 07:09:04.584561 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.227s user 0.000s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:09:04 dist-test-slave-69rg krb5kdc[30139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:09:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:09:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 07:09:04.954886 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38219
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:09:05.332376 30153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:09:05.332821 30153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:09:05.333009 30153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:09:05.341060 30153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:09:05.341325 30153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:09:05.341460 30153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:09:05.341578 30153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:09:05.341662 30153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:09:05.352227 30153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38219
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:09:05.353749 30153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:09:05.355182 30153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:09:05.370647 30159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:05.370591 30160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:05.370442 30162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:09:05.371774 30153 server_base.cc:1034] running on GCE node
I20260812 07:09:05.372706 30153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:09:05.374583 30153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:09:05.375931 30153 hybrid_clock.cc:648] HybridClock initialized: now 1786518545375867 us; error 61 us; skew 500 ppm
I20260812 07:09:05.376397 30153 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:09:05 dist-test-slave-69rg krb5kdc[30139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518545, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:09:05.381314 30153 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 07:09:05.383565 30153 webserver.cc:460] Webserver started at http://127.0.0.1:46777/ using document root <none> and password file <none>
I20260812 07:09:05.384539 30153 fs_manager.cc:359] Metadata directory not provided
I20260812 07:09:05.385007 30153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:09:05.386288 30153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:09:05.390753 30153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "e0f1d035570f4c0abe2919246f5be665"
format_stamp: "Formatted at 2026-08-12 07:09:05 on dist-test-slave-69rg"
server_key: "b4f8066ac8cc43240568f698b30e494f"
server_key_iv: "0c4caca0af6d12a70e464b5a7324711d"
server_key_version: "encryptionkey@0"
I20260812 07:09:05.391753 30153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "e0f1d035570f4c0abe2919246f5be665"
format_stamp: "Formatted at 2026-08-12 07:09:05 on dist-test-slave-69rg"
server_key: "b4f8066ac8cc43240568f698b30e494f"
server_key_iv: "0c4caca0af6d12a70e464b5a7324711d"
server_key_version: "encryptionkey@0"
I20260812 07:09:05.398942 30153 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260812 07:09:05.403415 30169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:09:05.405211 30153 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260812 07:09:05.405506 30153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "e0f1d035570f4c0abe2919246f5be665"
format_stamp: "Formatted at 2026-08-12 07:09:05 on dist-test-slave-69rg"
server_key: "b4f8066ac8cc43240568f698b30e494f"
server_key_iv: "0c4caca0af6d12a70e464b5a7324711d"
server_key_version: "encryptionkey@0"
I20260812 07:09:05.405772 30153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:09:05.425307 30153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:09:05.431185 30153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:09:05.431568 30153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:09:05.454697 30153 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38219
I20260812 07:09:05.454741 30227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38219 every 8 connection(s)
I20260812 07:09:05.456395 30153 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:09:05.462656 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 30153
I20260812 07:09:05.462917 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:09:05.462955 30228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:09:05.463845 6388 external_mini_cluster.cc:1428] Setting key 9ed22c40e2e6690e2f42dcb299246365
I20260812 07:09:05.481353 30228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0f1d035570f4c0abe2919246f5be665: Bootstrap starting.
Aug 12 07:09:05 dist-test-slave-69rg krb5kdc[30139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 07:09:05.485754 30228 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0f1d035570f4c0abe2919246f5be665: Neither blocks nor log segments found. Creating new log.
I20260812 07:09:05.487382 30228 log.cc:824] T 00000000000000000000000000000000 P e0f1d035570f4c0abe2919246f5be665: Log is configured to *not* fsync() on all Append() calls
I20260812 07:09:05.492175 30228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0f1d035570f4c0abe2919246f5be665: No bootstrap required, opened a new log
I20260812 07:09:05.496162 30231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:05.466320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40211 (local address 127.0.0.1:38219)
0812 07:09:05.468141 (+ 1821us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:05.468159 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:05.468213 (+ 54us) server_negotiation.cc:407] Connection header received
0812 07:09:05.470155 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:05.470193 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:05.470852 (+ 659us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:05.471409 (+ 557us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:05.473341 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:05.475398 (+ 2057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:05.477058 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:05.478417 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:05.478658 (+ 241us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:05.483781 (+ 5123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:05.483794 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:05.483815 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:05.483871 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:05.488939 (+ 5068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:05.490326 (+ 1387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:05.490337 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:05.490347 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:05.490578 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:05.491192 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:05.491197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:05.491201 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:05.491941 (+ 740us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:05.492312 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:05.493954 (+ 1642us) server_negotiation.cc:299] Negotiation successful
0812 07:09:05.494429 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1570,"thread_start_us":1163,"threads_started":1}
I20260812 07:09:05.502893 6388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:09:05.505216 30228 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0f1d035570f4c0abe2919246f5be665 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f1d035570f4c0abe2919246f5be665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38219 } }
I20260812 07:09:05.505746 30228 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0f1d035570f4c0abe2919246f5be665 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:09:05.505937 30228 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0f1d035570f4c0abe2919246f5be665 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0f1d035570f4c0abe2919246f5be665, State: Initialized, Role: FOLLOWER
I20260812 07:09:05.506742 30228 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0f1d035570f4c0abe2919246f5be665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f1d035570f4c0abe2919246f5be665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38219 } }
I20260812 07:09:05.507560 30228 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0f1d035570f4c0abe2919246f5be665 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:09:05.507743 30228 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0f1d035570f4c0abe2919246f5be665 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:09:05.507942 30228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0f1d035570f4c0abe2919246f5be665 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:09:05.511852 30228 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0f1d035570f4c0abe2919246f5be665 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f1d035570f4c0abe2919246f5be665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38219 } }
I20260812 07:09:05.512676 30228 leader_election.cc:302] T 00000000000000000000000000000000 P e0f1d035570f4c0abe2919246f5be665 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0f1d035570f4c0abe2919246f5be665; no voters:
I20260812 07:09:05.513566 30228 leader_election.cc:288] T 00000000000000000000000000000000 P e0f1d035570f4c0abe2919246f5be665 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:09:05.513793 30238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0f1d035570f4c0abe2919246f5be665 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:09:05.514546 30238 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0f1d035570f4c0abe2919246f5be665 [term 1 LEADER]: Becoming Leader. State: Replica: e0f1d035570f4c0abe2919246f5be665, State: Running, Role: LEADER
I20260812 07:09:05.515374 30238 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0f1d035570f4c0abe2919246f5be665 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f1d035570f4c0abe2919246f5be665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38219 } }
I20260812 07:09:05.519927 30240 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0f1d035570f4c0abe2919246f5be665 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0f1d035570f4c0abe2919246f5be665" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f1d035570f4c0abe2919246f5be665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38219 } } }
I20260812 07:09:05.520673 30240 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0f1d035570f4c0abe2919246f5be665 [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:05.522223 30238 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0f1d035570f4c0abe2919246f5be665 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0f1d035570f4c0abe2919246f5be665. Latest consensus state: current_term: 1 leader_uuid: "e0f1d035570f4c0abe2919246f5be665" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f1d035570f4c0abe2919246f5be665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38219 } } }
I20260812 07:09:05.523972 30238 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0f1d035570f4c0abe2919246f5be665 [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:05.529464 30228 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0f1d035570f4c0abe2919246f5be665 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:09:05.530068 30243 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:09:05.535795 30231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:05.508779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44699 (local address 127.0.0.1:38219)
0812 07:09:05.509045 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:05.509055 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:05.509900 (+ 845us) server_negotiation.cc:407] Connection header received
0812 07:09:05.509987 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:05.509998 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:05.510080 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:05.510198 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:05.512680 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:05.514207 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:05.516932 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:05.518238 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:05.518432 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:05.522948 (+ 4516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:05.522963 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:05.522972 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:05.523042 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:05.529365 (+ 6323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:05.530785 (+ 1420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:05.530792 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:05.530796 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:05.530975 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:05.531800 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:05.531810 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:05.531815 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:05.532431 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:05.532817 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:05.534864 (+ 2047us) server_negotiation.cc:299] Negotiation successful
0812 07:09:05.535167 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 07:09:05.538686 30243 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:09:05.558254 30243 catalog_manager.cc:1347] Generated new cluster ID: 0a83371abc8a41ec870232849f62116f
I20260812 07:09:05.558421 30243 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:09:05.578994 30243 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:09:05.580835 30243 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:09:05.590471 30243 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0f1d035570f4c0abe2919246f5be665: Generated new TSK 0
I20260812 07:09:05.591497 30243 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:09:05.605628 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 30153
I20260812 07:09:05.609000 30222 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:09:05.726320 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 30153
W20260812 07:09:05.755682 30137 connection.cc:535] client connection to 127.0.0.1:38219 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38219): Connection reset by peer (error 104)
2026-08-12T07:09:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:09:05 dist-test-slave-69rg krb5kdc[30262](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:09:05 dist-test-slave-69rg krb5kdc[30262](info): set up 2 sockets
Aug 12 07:09:05 dist-test-slave-69rg krb5kdc[30262](info): commencing operation
krb5kdc: starting...
W20260812 07:09:07.103987 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.327s user 0.002s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:09:07 dist-test-slave-69rg krb5kdc[30262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:09:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:09:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 07:09:07.497004 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37157
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:09:07.911309 30279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:09:07.911748 30279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:09:07.911919 30279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:09:07.919608 30279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:09:07.919811 30279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:09:07.919930 30279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:09:07.920048 30279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:09:07.920151 30279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:09:07.930855 30279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37157
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:09:07.932466 30279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:09:07.934175 30279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:09:07.949888 30288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:07.949820 30285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:07.950430 30286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:09:07.951473 30279 server_base.cc:1034] running on GCE node
I20260812 07:09:07.952464 30279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:09:07.954124 30279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:09:07.955454 30279 hybrid_clock.cc:648] HybridClock initialized: now 1786518547955398 us; error 54 us; skew 500 ppm
I20260812 07:09:07.955940 30279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:09:07 dist-test-slave-69rg krb5kdc[30262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:09:07.969754 30279 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 07:09:07.971853 30279 webserver.cc:460] Webserver started at http://127.0.0.1:36063/ using document root <none> and password file <none>
I20260812 07:09:07.972674 30279 fs_manager.cc:359] Metadata directory not provided
I20260812 07:09:07.972856 30279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:09:07.973212 30279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:09:07.976624 30279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "b7ea074f77994c168f6194c94af4f838"
format_stamp: "Formatted at 2026-08-12 07:09:07 on dist-test-slave-69rg"
server_key: "d8da85fae62dc1c1ac9fc9e4d142d51c"
server_key_iv: "d9b97c88b212f60b12464b426161d897"
server_key_version: "encryptionkey@0"
I20260812 07:09:07.977537 30279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "b7ea074f77994c168f6194c94af4f838"
format_stamp: "Formatted at 2026-08-12 07:09:07 on dist-test-slave-69rg"
server_key: "d8da85fae62dc1c1ac9fc9e4d142d51c"
server_key_iv: "d9b97c88b212f60b12464b426161d897"
server_key_version: "encryptionkey@0"
I20260812 07:09:07.983549 30279 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260812 07:09:07.988023 30295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:09:07.990250 30279 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 07:09:07.990541 30279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "b7ea074f77994c168f6194c94af4f838"
format_stamp: "Formatted at 2026-08-12 07:09:07 on dist-test-slave-69rg"
server_key: "d8da85fae62dc1c1ac9fc9e4d142d51c"
server_key_iv: "d9b97c88b212f60b12464b426161d897"
server_key_version: "encryptionkey@0"
I20260812 07:09:07.990819 30279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:09:08.011147 30279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:09:08.019073 30279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:09:08.019479 30279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:09:08.045672 30279 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37157
I20260812 07:09:08.045689 30348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37157 every 8 connection(s)
I20260812 07:09:08.047389 30279 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:09:08.051810 30349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:09:08.052660 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 30279
I20260812 07:09:08.052856 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:09:08.053422 6388 external_mini_cluster.cc:1428] Setting key f2f0afd0cc07ebeb86b5e3cefb68ff36
I20260812 07:09:08.072261 30349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7ea074f77994c168f6194c94af4f838: Bootstrap starting.
I20260812 07:09:08.072801 30352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:08.055615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55713 (local address 127.0.0.1:37157)
0812 07:09:08.056591 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:08.056604 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:08.056652 (+ 48us) server_negotiation.cc:407] Connection header received
0812 07:09:08.058426 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:08.058453 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:08.058952 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:08.059471 (+ 519us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:08.060428 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:08.062930 (+ 2502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:08.066209 (+ 3279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:08.067482 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:08.067727 (+ 245us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:08.069193 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:08.069210 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:08.069236 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:09:08.070053 (+ 817us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:08.070321 (+ 268us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:09:08.070367 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:08.070557 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:08.070921 (+ 364us) server_negotiation.cc:299] Negotiation successful
0812 07:09:08.071152 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":743,"thread_start_us":482,"threads_started":1}
I20260812 07:09:08.076567 30349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7ea074f77994c168f6194c94af4f838: Neither blocks nor log segments found. Creating new log.
I20260812 07:09:08.077973 30349 log.cc:824] T 00000000000000000000000000000000 P b7ea074f77994c168f6194c94af4f838: Log is configured to *not* fsync() on all Append() calls
I20260812 07:09:08.078311 6388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:09:08.082473 30349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7ea074f77994c168f6194c94af4f838: No bootstrap required, opened a new log
I20260812 07:09:08.093653 30352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:08.085422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47419 (local address 127.0.0.1:37157)
0812 07:09:08.085671 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:08.085679 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:08.086140 (+ 461us) server_negotiation.cc:407] Connection header received
0812 07:09:08.086336 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:08.086348 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:08.086453 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:08.086598 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:08.087213 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:08.088555 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:08.090270 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:08.091114 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:08.091273 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:08.092622 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:08.092631 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:08.092637 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:09:08.092674 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:08.092751 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:09:08.092776 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:08.092934 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:08.093184 (+ 250us) server_negotiation.cc:299] Negotiation successful
0812 07:09:08.093250 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 07:09:08.095160 30349 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7ea074f77994c168f6194c94af4f838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7ea074f77994c168f6194c94af4f838" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37157 } }
I20260812 07:09:08.095678 30349 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7ea074f77994c168f6194c94af4f838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:09:08.095810 30349 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7ea074f77994c168f6194c94af4f838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7ea074f77994c168f6194c94af4f838, State: Initialized, Role: FOLLOWER
I20260812 07:09:08.096484 30349 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7ea074f77994c168f6194c94af4f838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7ea074f77994c168f6194c94af4f838" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37157 } }
I20260812 07:09:08.096732 30349 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7ea074f77994c168f6194c94af4f838 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:09:08.096837 30349 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7ea074f77994c168f6194c94af4f838 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:09:08.096983 30349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7ea074f77994c168f6194c94af4f838 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:09:08.099870 30349 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7ea074f77994c168f6194c94af4f838 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7ea074f77994c168f6194c94af4f838" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37157 } }
I20260812 07:09:08.100551 30349 leader_election.cc:302] T 00000000000000000000000000000000 P b7ea074f77994c168f6194c94af4f838 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7ea074f77994c168f6194c94af4f838; no voters:
I20260812 07:09:08.101516 30349 leader_election.cc:288] T 00000000000000000000000000000000 P b7ea074f77994c168f6194c94af4f838 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:09:08.101653 30360 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7ea074f77994c168f6194c94af4f838 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:09:08.102958 30360 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7ea074f77994c168f6194c94af4f838 [term 1 LEADER]: Becoming Leader. State: Replica: b7ea074f77994c168f6194c94af4f838, State: Running, Role: LEADER
I20260812 07:09:08.103840 30360 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7ea074f77994c168f6194c94af4f838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7ea074f77994c168f6194c94af4f838" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37157 } }
I20260812 07:09:08.105912 30349 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7ea074f77994c168f6194c94af4f838 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:09:08.107192 30362 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7ea074f77994c168f6194c94af4f838 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7ea074f77994c168f6194c94af4f838" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7ea074f77994c168f6194c94af4f838" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37157 } } }
I20260812 07:09:08.107261 30361 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7ea074f77994c168f6194c94af4f838 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7ea074f77994c168f6194c94af4f838. Latest consensus state: current_term: 1 leader_uuid: "b7ea074f77994c168f6194c94af4f838" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7ea074f77994c168f6194c94af4f838" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37157 } } }
I20260812 07:09:08.107735 30362 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7ea074f77994c168f6194c94af4f838 [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:08.107743 30361 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7ea074f77994c168f6194c94af4f838 [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:08.110647 30365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:09:08.116691 30365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:09:08.128783 30365 catalog_manager.cc:1347] Generated new cluster ID: a3ef1467bf024631a81f8f7baf97419d
I20260812 07:09:08.128984 30365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:09:08.144876 30365 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:09:08.146967 30365 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:09:08.157836 30365 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7ea074f77994c168f6194c94af4f838: Generated new TSK 0
I20260812 07:09:08.159106 30365 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:09:08.179972 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 30279
W20260812 07:09:08.181427 30344 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55713
W20260812 07:09:08.182547 6388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 30279: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 07:09:08.182808 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 30279
W20260812 07:09:08.209342 30260 connection.cc:535] client connection to 127.0.0.1:37157 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37157): Connection reset by peer (error 104)
2026-08-12T07:09:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2453 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:09:08 dist-test-slave-69rg krb5kdc[30381](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:09:08 dist-test-slave-69rg krb5kdc[30381](info): set up 2 sockets
Aug 12 07:09:08 dist-test-slave-69rg krb5kdc[30381](info): commencing operation
krb5kdc: starting...
W20260812 07:09:09.574221 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.343s user 0.000s sys 0.053s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:09:09 dist-test-slave-69rg krb5kdc[30381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:09:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:09:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 07:09:09.974819 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33457
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:09:10.399039 30395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:09:10.399410 30395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:09:10.399567 30395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:09:10.409782 30395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:09:10.409971 30395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:09:10.410025 30395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:09:10.410199 30395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:09:10.410276 30395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:09:10.426265 30395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33457
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:09:10.427966 30395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:09:10.431075 30395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:09:10.450446 30401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:10.450500 30406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:10.450922 30402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:09:10.451022 30395 server_base.cc:1034] running on GCE node
I20260812 07:09:10.452380 30395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:09:10.454123 30395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:09:10.455432 30395 hybrid_clock.cc:648] HybridClock initialized: now 1786518550455360 us; error 74 us; skew 500 ppm
I20260812 07:09:10.455878 30395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:09:10 dist-test-slave-69rg krb5kdc[30381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:09:10.461324 30395 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 07:09:10.463549 30395 webserver.cc:460] Webserver started at http://127.0.0.1:34627/ using document root <none> and password file <none>
I20260812 07:09:10.464618 30395 fs_manager.cc:359] Metadata directory not provided
I20260812 07:09:10.464756 30395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:09:10.465150 30395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:09:10.469532 30395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "8c91b1b5ed98434f80b857adb95963ae"
format_stamp: "Formatted at 2026-08-12 07:09:10 on dist-test-slave-69rg"
server_key: "b6b5e1f2536b7371914b001f2032f21b"
server_key_iv: "745a502ff30abcb3dcee1d199c5f17e5"
server_key_version: "encryptionkey@0"
I20260812 07:09:10.471170 30395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "8c91b1b5ed98434f80b857adb95963ae"
format_stamp: "Formatted at 2026-08-12 07:09:10 on dist-test-slave-69rg"
server_key: "b6b5e1f2536b7371914b001f2032f21b"
server_key_iv: "745a502ff30abcb3dcee1d199c5f17e5"
server_key_version: "encryptionkey@0"
I20260812 07:09:10.478346 30395 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260812 07:09:10.482880 30414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:09:10.485267 30395 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 07:09:10.485718 30395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "8c91b1b5ed98434f80b857adb95963ae"
format_stamp: "Formatted at 2026-08-12 07:09:10 on dist-test-slave-69rg"
server_key: "b6b5e1f2536b7371914b001f2032f21b"
server_key_iv: "745a502ff30abcb3dcee1d199c5f17e5"
server_key_version: "encryptionkey@0"
I20260812 07:09:10.486038 30395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:09:10.494155 30395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:09:10.500911 30395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:09:10.501336 30395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:09:10.531456 30395 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33457
I20260812 07:09:10.531466 30468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33457 every 8 connection(s)
I20260812 07:09:10.533995 30395 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:09:10.538803 30469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:09:10.541739 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 30395
I20260812 07:09:10.541977 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:09:10.542536 6388 external_mini_cluster.cc:1428] Setting key 9c9fcbd87941595bbb612a350a18d831
I20260812 07:09:10.552366 30472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:10.545051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49939 (local address 127.0.0.1:33457)
0812 07:09:10.546051 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:10.546064 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:10.546110 (+ 46us) server_negotiation.cc:407] Connection header received
0812 07:09:10.548302 (+ 2192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:10.548333 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:10.548747 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:10.549230 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:10.550481 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:10.550488 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:10.550506 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:09:10.550523 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:10.550594 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:09:10.550624 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:10.550940 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:10.551106 (+ 166us) server_negotiation.cc:299] Negotiation successful
0812 07:09:10.551338 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":456,"threads_started":1}
I20260812 07:09:10.553332 30469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c91b1b5ed98434f80b857adb95963ae: Bootstrap starting.
I20260812 07:09:10.556922 6388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:09:10.557545 30469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c91b1b5ed98434f80b857adb95963ae: Neither blocks nor log segments found. Creating new log.
I20260812 07:09:10.559415 30469 log.cc:824] T 00000000000000000000000000000000 P 8c91b1b5ed98434f80b857adb95963ae: Log is configured to *not* fsync() on all Append() calls
I20260812 07:09:10.563601 30469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c91b1b5ed98434f80b857adb95963ae: No bootstrap required, opened a new log
I20260812 07:09:10.566938 30472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:10.563356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46585 (local address 127.0.0.1:33457)
0812 07:09:10.563703 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:10.563710 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:10.564189 (+ 479us) server_negotiation.cc:407] Connection header received
0812 07:09:10.564422 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:10.564427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:10.564488 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:10.564584 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:10.565813 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:10.565914 (+ 101us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:10.565918 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:09:10.565942 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:10.565990 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:09:10.566021 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:10.566136 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:10.566547 (+ 411us) server_negotiation.cc:299] Negotiation successful
0812 07:09:10.566637 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260812 07:09:10.573701 30469 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c91b1b5ed98434f80b857adb95963ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c91b1b5ed98434f80b857adb95963ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33457 } }
I20260812 07:09:10.574270 30469 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c91b1b5ed98434f80b857adb95963ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:09:10.574378 30469 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c91b1b5ed98434f80b857adb95963ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c91b1b5ed98434f80b857adb95963ae, State: Initialized, Role: FOLLOWER
I20260812 07:09:10.575068 30469 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c91b1b5ed98434f80b857adb95963ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c91b1b5ed98434f80b857adb95963ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33457 } }
I20260812 07:09:10.575371 30469 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c91b1b5ed98434f80b857adb95963ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:09:10.575682 30469 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c91b1b5ed98434f80b857adb95963ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:09:10.575927 30469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c91b1b5ed98434f80b857adb95963ae [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:09:10.578536 30469 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c91b1b5ed98434f80b857adb95963ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c91b1b5ed98434f80b857adb95963ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33457 } }
I20260812 07:09:10.579144 30469 leader_election.cc:302] T 00000000000000000000000000000000 P 8c91b1b5ed98434f80b857adb95963ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c91b1b5ed98434f80b857adb95963ae; no voters:
I20260812 07:09:10.579883 30469 leader_election.cc:288] T 00000000000000000000000000000000 P 8c91b1b5ed98434f80b857adb95963ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:09:10.580060 30480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c91b1b5ed98434f80b857adb95963ae [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:09:10.580835 30480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c91b1b5ed98434f80b857adb95963ae [term 1 LEADER]: Becoming Leader. State: Replica: 8c91b1b5ed98434f80b857adb95963ae, State: Running, Role: LEADER
I20260812 07:09:10.581439 30480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c91b1b5ed98434f80b857adb95963ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c91b1b5ed98434f80b857adb95963ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33457 } }
I20260812 07:09:10.582252 30469 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c91b1b5ed98434f80b857adb95963ae [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:09:10.585050 30481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c91b1b5ed98434f80b857adb95963ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c91b1b5ed98434f80b857adb95963ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c91b1b5ed98434f80b857adb95963ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33457 } } }
I20260812 07:09:10.584895 30482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c91b1b5ed98434f80b857adb95963ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c91b1b5ed98434f80b857adb95963ae. Latest consensus state: current_term: 1 leader_uuid: "8c91b1b5ed98434f80b857adb95963ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c91b1b5ed98434f80b857adb95963ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33457 } } }
I20260812 07:09:10.585417 30482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c91b1b5ed98434f80b857adb95963ae [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:10.585417 30481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c91b1b5ed98434f80b857adb95963ae [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:10.593334 30490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:09:10.599884 30490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:09:10.609575 30490 catalog_manager.cc:1347] Generated new cluster ID: 1f30c3256ba94538b388c528196de0e0
I20260812 07:09:10.609712 30490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:09:10.634538 30490 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:09:10.635450 30490 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:09:10.653930 30490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c91b1b5ed98434f80b857adb95963ae: Generated new TSK 0
I20260812 07:09:10.655030 30490 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:09:10.664976 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 30395
W20260812 07:09:10.666389 30464 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49939
W20260812 07:09:10.667364 6388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 30395: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 07:09:10.667589 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 30395
2026-08-12T07:09:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2483 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:09:10 dist-test-slave-69rg krb5kdc[30503](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:09:10 dist-test-slave-69rg krb5kdc[30503](info): set up 2 sockets
Aug 12 07:09:10 dist-test-slave-69rg krb5kdc[30503](info): commencing operation
krb5kdc: starting...
W20260812 07:09:11.998764 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.285s user 0.005s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:09:12 dist-test-slave-69rg krb5kdc[30503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:09:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:09:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 07:09:12.413313 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45119
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:09:12.829267 30518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:09:12.829592 30518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:09:12.829766 30518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:09:12.838757 30518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:09:12.838902 30518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:09:12.838995 30518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:09:12.839089 30518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:09:12.839171 30518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:09:12.850801 30518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:09:12.852386 30518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:09:12.853916 30518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:09:12.872761 30527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:12.872668 30530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:12.872741 30528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:09:12.875914 30518 server_base.cc:1034] running on GCE node
I20260812 07:09:12.877038 30518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:09:12.879923 30518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:09:12.881327 30518 hybrid_clock.cc:648] HybridClock initialized: now 1786518552881236 us; error 92 us; skew 500 ppm
I20260812 07:09:12.882580 30518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:09:12 dist-test-slave-69rg krb5kdc[30503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:09:12.887387 30518 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 07:09:12.889741 30518 webserver.cc:460] Webserver started at http://127.0.0.1:36401/ using document root <none> and password file <none>
I20260812 07:09:12.890692 30518 fs_manager.cc:359] Metadata directory not provided
I20260812 07:09:12.890888 30518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:09:12.891247 30518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:09:12.894733 30518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "9c58d4398236494ea29335c7ab73610b"
format_stamp: "Formatted at 2026-08-12 07:09:12 on dist-test-slave-69rg"
server_key: "214bd003c634807e1577d88b2a2d7a6a"
server_key_iv: "3b1db787214a312fddcc176bff87d319"
server_key_version: "encryptionkey@0"
I20260812 07:09:12.895700 30518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "9c58d4398236494ea29335c7ab73610b"
format_stamp: "Formatted at 2026-08-12 07:09:12 on dist-test-slave-69rg"
server_key: "214bd003c634807e1577d88b2a2d7a6a"
server_key_iv: "3b1db787214a312fddcc176bff87d319"
server_key_version: "encryptionkey@0"
I20260812 07:09:12.902524 30518 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260812 07:09:12.908850 30538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:09:12.910594 30518 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 07:09:12.910809 30518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "9c58d4398236494ea29335c7ab73610b"
format_stamp: "Formatted at 2026-08-12 07:09:12 on dist-test-slave-69rg"
server_key: "214bd003c634807e1577d88b2a2d7a6a"
server_key_iv: "3b1db787214a312fddcc176bff87d319"
server_key_version: "encryptionkey@0"
I20260812 07:09:12.911031 30518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:09:12.931668 30518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:09:12.937884 30518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:09:12.938223 30518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:09:12.960354 30518 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45119
I20260812 07:09:12.960386 30593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45119 every 8 connection(s)
I20260812 07:09:12.961911 30518 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:09:12.965168 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 30518
I20260812 07:09:12.965405 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:09:12.966109 6388 external_mini_cluster.cc:1428] Setting key 0b61fa29ec1eaa543f5df2a100075040
I20260812 07:09:12.967128 30594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:09:12.983433 30594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c58d4398236494ea29335c7ab73610b: Bootstrap starting.
I20260812 07:09:12.987927 30594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c58d4398236494ea29335c7ab73610b: Neither blocks nor log segments found. Creating new log.
Aug 12 07:09:12 dist-test-slave-69rg krb5kdc[30503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 07:09:12.989408 30594 log.cc:824] T 00000000000000000000000000000000 P 9c58d4398236494ea29335c7ab73610b: Log is configured to *not* fsync() on all Append() calls
I20260812 07:09:12.993631 30594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c58d4398236494ea29335c7ab73610b: No bootstrap required, opened a new log
I20260812 07:09:13.002656 30597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:12.968304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46215 (local address 127.0.0.1:45119)
0812 07:09:12.969217 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:12.969230 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:12.969314 (+ 84us) server_negotiation.cc:407] Connection header received
0812 07:09:12.971032 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:12.971057 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:12.971550 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:12.972006 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:12.974020 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:12.976387 (+ 2367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:12.978364 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:12.979534 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:12.979747 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:12.990692 (+ 10945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:12.990705 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:12.990724 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:12.990771 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:12.995280 (+ 4509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:12.996781 (+ 1501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:12.996793 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:12.996803 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:12.997234 (+ 431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:12.997888 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:12.997893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:12.997897 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:12.998780 (+ 883us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:12.999332 (+ 552us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:12.999938 (+ 606us) server_negotiation.cc:299] Negotiation successful
0812 07:09:13.000516 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":453,"threads_started":1}
I20260812 07:09:13.007948 30594 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c58d4398236494ea29335c7ab73610b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c58d4398236494ea29335c7ab73610b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45119 } }
I20260812 07:09:13.008750 30594 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c58d4398236494ea29335c7ab73610b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:09:13.008937 30594 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c58d4398236494ea29335c7ab73610b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c58d4398236494ea29335c7ab73610b, State: Initialized, Role: FOLLOWER
I20260812 07:09:13.009240 6388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:09:13.009719 30594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c58d4398236494ea29335c7ab73610b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c58d4398236494ea29335c7ab73610b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45119 } }
I20260812 07:09:13.010069 30594 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c58d4398236494ea29335c7ab73610b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:09:13.010249 30594 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c58d4398236494ea29335c7ab73610b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:09:13.010452 30594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c58d4398236494ea29335c7ab73610b [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:09:13.013675 30594 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c58d4398236494ea29335c7ab73610b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c58d4398236494ea29335c7ab73610b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45119 } }
I20260812 07:09:13.014457 30594 leader_election.cc:302] T 00000000000000000000000000000000 P 9c58d4398236494ea29335c7ab73610b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c58d4398236494ea29335c7ab73610b; no voters:
I20260812 07:09:13.015282 30594 leader_election.cc:288] T 00000000000000000000000000000000 P 9c58d4398236494ea29335c7ab73610b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:09:13.015743 30603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c58d4398236494ea29335c7ab73610b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:09:13.018148 30603 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c58d4398236494ea29335c7ab73610b [term 1 LEADER]: Becoming Leader. State: Replica: 9c58d4398236494ea29335c7ab73610b, State: Running, Role: LEADER
I20260812 07:09:13.019068 30603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c58d4398236494ea29335c7ab73610b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c58d4398236494ea29335c7ab73610b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45119 } }
I20260812 07:09:13.019364 30594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c58d4398236494ea29335c7ab73610b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:09:13.024369 30604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c58d4398236494ea29335c7ab73610b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c58d4398236494ea29335c7ab73610b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c58d4398236494ea29335c7ab73610b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45119 } } }
I20260812 07:09:13.024883 30604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c58d4398236494ea29335c7ab73610b [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:13.025600 30607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c58d4398236494ea29335c7ab73610b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c58d4398236494ea29335c7ab73610b. Latest consensus state: current_term: 1 leader_uuid: "9c58d4398236494ea29335c7ab73610b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c58d4398236494ea29335c7ab73610b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45119 } } }
I20260812 07:09:13.026218 30607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c58d4398236494ea29335c7ab73610b [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:13.028905 30615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:09:13.035470 30615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:09:13.039976 30597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:13.015874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42465 (local address 127.0.0.1:45119)
0812 07:09:13.016288 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:13.016299 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:13.019626 (+ 3327us) server_negotiation.cc:407] Connection header received
0812 07:09:13.019882 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:13.019888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:13.019972 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:13.020094 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:13.022871 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:13.024249 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:13.026254 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:13.026810 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:13.026974 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:13.030760 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:13.030769 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:13.030773 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:13.030823 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:13.035304 (+ 4481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:13.036407 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:13.036418 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:13.036424 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:13.036683 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:13.037184 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:13.037191 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:13.037200 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:13.037636 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:13.037961 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:13.039329 (+ 1368us) server_negotiation.cc:299] Negotiation successful
0812 07:09:13.039572 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":153}
I20260812 07:09:13.049002 30615 catalog_manager.cc:1347] Generated new cluster ID: 19015634689b4fa49092d8eca4fcf216
I20260812 07:09:13.049155 30615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:09:13.090137 30615 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:09:13.092264 30615 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:09:13.103618 30615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c58d4398236494ea29335c7ab73610b: Generated new TSK 0
I20260812 07:09:13.104938 30615 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:09:13.125095 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 30518
I20260812 07:09:13.126912 30589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:09:13.228122 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 30518
2026-08-12T07:09:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2560 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:09:13 dist-test-slave-69rg krb5kdc[30631](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:09:13 dist-test-slave-69rg krb5kdc[30631](info): set up 2 sockets
Aug 12 07:09:13 dist-test-slave-69rg krb5kdc[30631](info): commencing operation
krb5kdc: starting...
W20260812 07:09:14.549460 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.276s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:09:14 dist-test-slave-69rg krb5kdc[30631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:09:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:09:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 07:09:14.931826 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35885
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:09:15.312661 30645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:09:15.313000 30645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:09:15.313189 30645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:09:15.320956 30645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:09:15.321098 30645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:09:15.321218 30645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:09:15.321333 30645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:09:15.321408 30645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:09:15.331557 30645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:09:15.332930 30645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:09:15.334290 30645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:09:15.347147 30652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:15.347481 30653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:15.347113 30655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:09:15.349175 30645 server_base.cc:1034] running on GCE node
I20260812 07:09:15.350221 30645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:09:15.351859 30645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:09:15.353120 30645 hybrid_clock.cc:648] HybridClock initialized: now 1786518555353043 us; error 66 us; skew 500 ppm
I20260812 07:09:15.353519 30645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:09:15 dist-test-slave-69rg krb5kdc[30631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:09:15.357259 30645 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 07:09:15.359273 30645 webserver.cc:460] Webserver started at http://127.0.0.1:39183/ using document root <none> and password file <none>
I20260812 07:09:15.360136 30645 fs_manager.cc:359] Metadata directory not provided
I20260812 07:09:15.360297 30645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:09:15.360682 30645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:09:15.364838 30645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "11c5d7c19681496c8cb51e7a228c0c28"
format_stamp: "Formatted at 2026-08-12 07:09:15 on dist-test-slave-69rg"
server_key: "e8865db50a10728a92e62c904d648097"
server_key_iv: "67192ee77b6c3e556846fc2944645f36"
server_key_version: "encryptionkey@0"
I20260812 07:09:15.365960 30645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "11c5d7c19681496c8cb51e7a228c0c28"
format_stamp: "Formatted at 2026-08-12 07:09:15 on dist-test-slave-69rg"
server_key: "e8865db50a10728a92e62c904d648097"
server_key_iv: "67192ee77b6c3e556846fc2944645f36"
server_key_version: "encryptionkey@0"
I20260812 07:09:15.372813 30645 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260812 07:09:15.378176 30664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:09:15.380409 30645 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 07:09:15.380717 30645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "11c5d7c19681496c8cb51e7a228c0c28"
format_stamp: "Formatted at 2026-08-12 07:09:15 on dist-test-slave-69rg"
server_key: "e8865db50a10728a92e62c904d648097"
server_key_iv: "67192ee77b6c3e556846fc2944645f36"
server_key_version: "encryptionkey@0"
I20260812 07:09:15.380963 30645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:09:15.414661 30645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:09:15.419962 30645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:09:15.420337 30645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:09:15.442088 30645 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35885
I20260812 07:09:15.442142 30719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35885 every 8 connection(s)
I20260812 07:09:15.443701 30645 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:09:15.448132 30720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:09:15.450273 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 30645
I20260812 07:09:15.450492 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:09:15.451009 6388 external_mini_cluster.cc:1428] Setting key c2ac779f203a58a0b8cc06ba674eaabd
I20260812 07:09:15.463110 30720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11c5d7c19681496c8cb51e7a228c0c28: Bootstrap starting.
I20260812 07:09:15.465690 30723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:15.453105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39145 (local address 127.0.0.1:35885)
0812 07:09:15.454210 (+ 1105us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:15.454268 (+ 58us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:15.454319 (+ 51us) server_negotiation.cc:407] Connection header received
0812 07:09:15.456414 (+ 2095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:15.456441 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:15.456926 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:15.457366 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:15.458015 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:15.459354 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:15.460773 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:15.461506 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:15.461742 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:15.463140 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:15.463149 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:15.463238 (+ 89us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:09:15.463853 (+ 615us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:15.463945 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:09:15.463975 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:15.464127 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:15.464371 (+ 244us) server_negotiation.cc:299] Negotiation successful
0812 07:09:15.464688 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":838,"thread_start_us":442,"threads_started":1}
I20260812 07:09:15.467020 30720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11c5d7c19681496c8cb51e7a228c0c28: Neither blocks nor log segments found. Creating new log.
I20260812 07:09:15.468322 30720 log.cc:824] T 00000000000000000000000000000000 P 11c5d7c19681496c8cb51e7a228c0c28: Log is configured to *not* fsync() on all Append() calls
I20260812 07:09:15.470711 6388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:09:15.472183 30720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11c5d7c19681496c8cb51e7a228c0c28: No bootstrap required, opened a new log
I20260812 07:09:15.484117 30723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:15.475611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54605 (local address 127.0.0.1:35885)
0812 07:09:15.475858 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:15.475868 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:15.476330 (+ 462us) server_negotiation.cc:407] Connection header received
0812 07:09:15.476573 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:15.476578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:15.476647 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:15.476766 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:15.477565 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:15.478622 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:15.480938 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:15.481453 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:15.481587 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:15.483074 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:15.483080 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:15.483084 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:09:15.483107 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:15.483155 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:09:15.483173 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:15.483309 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:15.483559 (+ 250us) server_negotiation.cc:299] Negotiation successful
0812 07:09:15.483652 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 07:09:15.483909 30720 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11c5d7c19681496c8cb51e7a228c0c28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c5d7c19681496c8cb51e7a228c0c28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35885 } }
I20260812 07:09:15.484495 30720 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11c5d7c19681496c8cb51e7a228c0c28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:09:15.484601 30720 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11c5d7c19681496c8cb51e7a228c0c28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c5d7c19681496c8cb51e7a228c0c28, State: Initialized, Role: FOLLOWER
I20260812 07:09:15.485324 30720 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11c5d7c19681496c8cb51e7a228c0c28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c5d7c19681496c8cb51e7a228c0c28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35885 } }
I20260812 07:09:15.485751 30720 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11c5d7c19681496c8cb51e7a228c0c28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:09:15.486012 30720 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11c5d7c19681496c8cb51e7a228c0c28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:09:15.486236 30720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11c5d7c19681496c8cb51e7a228c0c28 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:09:15.488961 30720 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11c5d7c19681496c8cb51e7a228c0c28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c5d7c19681496c8cb51e7a228c0c28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35885 } }
I20260812 07:09:15.489666 30720 leader_election.cc:302] T 00000000000000000000000000000000 P 11c5d7c19681496c8cb51e7a228c0c28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11c5d7c19681496c8cb51e7a228c0c28; no voters:
I20260812 07:09:15.490448 30720 leader_election.cc:288] T 00000000000000000000000000000000 P 11c5d7c19681496c8cb51e7a228c0c28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:09:15.490613 30730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11c5d7c19681496c8cb51e7a228c0c28 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:09:15.492129 30730 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11c5d7c19681496c8cb51e7a228c0c28 [term 1 LEADER]: Becoming Leader. State: Replica: 11c5d7c19681496c8cb51e7a228c0c28, State: Running, Role: LEADER
I20260812 07:09:15.492798 30730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11c5d7c19681496c8cb51e7a228c0c28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c5d7c19681496c8cb51e7a228c0c28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35885 } }
I20260812 07:09:15.492964 30720 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11c5d7c19681496c8cb51e7a228c0c28 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:09:15.496249 30731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11c5d7c19681496c8cb51e7a228c0c28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11c5d7c19681496c8cb51e7a228c0c28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c5d7c19681496c8cb51e7a228c0c28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35885 } } }
I20260812 07:09:15.496212 30732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11c5d7c19681496c8cb51e7a228c0c28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11c5d7c19681496c8cb51e7a228c0c28. Latest consensus state: current_term: 1 leader_uuid: "11c5d7c19681496c8cb51e7a228c0c28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c5d7c19681496c8cb51e7a228c0c28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35885 } } }
I20260812 07:09:15.496807 30731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11c5d7c19681496c8cb51e7a228c0c28 [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:15.496868 30732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11c5d7c19681496c8cb51e7a228c0c28 [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:15.499455 30738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:09:15.505350 30738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:09:15.516106 30738 catalog_manager.cc:1347] Generated new cluster ID: 7c5bcb690c5145408441db44e5faa62a
I20260812 07:09:15.516330 30738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:09:15.527518 30738 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:09:15.528949 30738 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:09:15.536283 30738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11c5d7c19681496c8cb51e7a228c0c28: Generated new TSK 0
I20260812 07:09:15.537194 30738 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:09:15.548123 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 30645
W20260812 07:09:15.549542 30715 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39145
W20260812 07:09:15.550536 6388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 30645: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 07:09:15.550726 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 30645
2026-08-12T07:09:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2317 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:09:15 dist-test-slave-69rg krb5kdc[30755](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:09:15 dist-test-slave-69rg krb5kdc[30755](info): set up 2 sockets
Aug 12 07:09:15 dist-test-slave-69rg krb5kdc[30755](info): commencing operation
krb5kdc: starting...
W20260812 07:09:16.853014 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.261s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:09:17 dist-test-slave-69rg krb5kdc[30755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:09:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:09:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 07:09:17.268080 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41151
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:09:17.675623 30771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:09:17.675974 30771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:09:17.676172 30771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:09:17.685637 30771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:09:17.685791 30771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:09:17.685916 30771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:09:17.686046 30771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:09:17.686123 30771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:09:17.698251 30771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:09:17.699955 30771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:09:17.701557 30771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:09:17.718420 30777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:17.718407 30776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:09:17.719820 30771 server_base.cc:1034] running on GCE node
W20260812 07:09:17.719898 30779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:09:17.721047 30771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:09:17.722791 30771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:09:17.724072 30771 hybrid_clock.cc:648] HybridClock initialized: now 1786518557724004 us; error 53 us; skew 500 ppm
I20260812 07:09:17.724525 30771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:09:17 dist-test-slave-69rg krb5kdc[30755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:09:17.728368 30771 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 07:09:17.730289 30771 webserver.cc:460] Webserver started at http://127.0.0.1:46633/ using document root <none> and password file <none>
I20260812 07:09:17.731077 30771 fs_manager.cc:359] Metadata directory not provided
I20260812 07:09:17.731189 30771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:09:17.731539 30771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:09:17.734898 30771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "fa4766585198468e9d5f6f26fb6a3d31"
format_stamp: "Formatted at 2026-08-12 07:09:17 on dist-test-slave-69rg"
server_key: "142e6261ae1fbd1fb5509bde325c5ea8"
server_key_iv: "f6d3b9d807b9175687e5a841b0859d07"
server_key_version: "encryptionkey@0"
I20260812 07:09:17.735773 30771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "fa4766585198468e9d5f6f26fb6a3d31"
format_stamp: "Formatted at 2026-08-12 07:09:17 on dist-test-slave-69rg"
server_key: "142e6261ae1fbd1fb5509bde325c5ea8"
server_key_iv: "f6d3b9d807b9175687e5a841b0859d07"
server_key_version: "encryptionkey@0"
I20260812 07:09:17.741933 30771 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260812 07:09:17.745931 30786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:09:17.747550 30771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 07:09:17.747802 30771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "fa4766585198468e9d5f6f26fb6a3d31"
format_stamp: "Formatted at 2026-08-12 07:09:17 on dist-test-slave-69rg"
server_key: "142e6261ae1fbd1fb5509bde325c5ea8"
server_key_iv: "f6d3b9d807b9175687e5a841b0859d07"
server_key_version: "encryptionkey@0"
I20260812 07:09:17.748047 30771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:09:17.754524 30771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:09:17.759265 30771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:09:17.759637 30771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:09:17.780998 30771 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41151
I20260812 07:09:17.781061 30844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41151 every 8 connection(s)
I20260812 07:09:17.782718 30771 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:09:17.787389 30845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:09:17.790410 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 30771
I20260812 07:09:17.790616 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:09:17.791188 6388 external_mini_cluster.cc:1428] Setting key 3e04484b843597359f7ab1f418767482
I20260812 07:09:17.800034 30848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:17.793281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41623 (local address 127.0.0.1:41151)
0812 07:09:17.794218 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:17.794235 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:17.794324 (+ 89us) server_negotiation.cc:407] Connection header received
0812 07:09:17.796045 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:17.796067 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:17.796407 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:17.796814 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:17.797976 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:17.797986 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:17.797999 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:09:17.798013 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:17.798098 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:09:17.798125 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:17.798411 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:17.798563 (+ 152us) server_negotiation.cc:299] Negotiation successful
0812 07:09:17.798752 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":458,"threads_started":1}
I20260812 07:09:17.801101 30845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa4766585198468e9d5f6f26fb6a3d31: Bootstrap starting.
I20260812 07:09:17.804152 6388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:09:17.804632 30845 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa4766585198468e9d5f6f26fb6a3d31: Neither blocks nor log segments found. Creating new log.
I20260812 07:09:17.805676 30845 log.cc:824] T 00000000000000000000000000000000 P fa4766585198468e9d5f6f26fb6a3d31: Log is configured to *not* fsync() on all Append() calls
I20260812 07:09:17.808818 30845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa4766585198468e9d5f6f26fb6a3d31: No bootstrap required, opened a new log
I20260812 07:09:17.811714 30848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:17.808834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33681 (local address 127.0.0.1:41151)
0812 07:09:17.809059 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:17.809065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:17.809473 (+ 408us) server_negotiation.cc:407] Connection header received
0812 07:09:17.809668 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:17.809671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:17.809722 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:17.809807 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:17.810856 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:17.810861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:17.810864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:09:17.810879 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:17.810923 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:09:17.810938 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:17.811043 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:17.811248 (+ 205us) server_negotiation.cc:299] Negotiation successful
0812 07:09:17.811356 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 07:09:17.817471 30845 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa4766585198468e9d5f6f26fb6a3d31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4766585198468e9d5f6f26fb6a3d31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41151 } }
I20260812 07:09:17.817973 30845 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa4766585198468e9d5f6f26fb6a3d31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:09:17.818101 30845 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa4766585198468e9d5f6f26fb6a3d31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa4766585198468e9d5f6f26fb6a3d31, State: Initialized, Role: FOLLOWER
I20260812 07:09:17.818708 30845 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa4766585198468e9d5f6f26fb6a3d31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4766585198468e9d5f6f26fb6a3d31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41151 } }
I20260812 07:09:17.818922 30845 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa4766585198468e9d5f6f26fb6a3d31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:09:17.819065 30845 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa4766585198468e9d5f6f26fb6a3d31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:09:17.819252 30845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa4766585198468e9d5f6f26fb6a3d31 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:09:17.821733 30845 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa4766585198468e9d5f6f26fb6a3d31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4766585198468e9d5f6f26fb6a3d31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41151 } }
I20260812 07:09:17.822388 30845 leader_election.cc:302] T 00000000000000000000000000000000 P fa4766585198468e9d5f6f26fb6a3d31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa4766585198468e9d5f6f26fb6a3d31; no voters:
I20260812 07:09:17.823047 30845 leader_election.cc:288] T 00000000000000000000000000000000 P fa4766585198468e9d5f6f26fb6a3d31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:09:17.823233 30855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa4766585198468e9d5f6f26fb6a3d31 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:09:17.824079 30855 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa4766585198468e9d5f6f26fb6a3d31 [term 1 LEADER]: Becoming Leader. State: Replica: fa4766585198468e9d5f6f26fb6a3d31, State: Running, Role: LEADER
I20260812 07:09:17.824724 30855 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa4766585198468e9d5f6f26fb6a3d31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4766585198468e9d5f6f26fb6a3d31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41151 } }
I20260812 07:09:17.825527 30845 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa4766585198468e9d5f6f26fb6a3d31 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:09:17.828094 30856 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa4766585198468e9d5f6f26fb6a3d31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa4766585198468e9d5f6f26fb6a3d31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4766585198468e9d5f6f26fb6a3d31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41151 } } }
I20260812 07:09:17.828596 30856 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa4766585198468e9d5f6f26fb6a3d31 [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:17.830169 30858 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa4766585198468e9d5f6f26fb6a3d31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa4766585198468e9d5f6f26fb6a3d31. Latest consensus state: current_term: 1 leader_uuid: "fa4766585198468e9d5f6f26fb6a3d31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4766585198468e9d5f6f26fb6a3d31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41151 } } }
I20260812 07:09:17.830653 30858 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa4766585198468e9d5f6f26fb6a3d31 [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:17.830689 30866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:09:17.836834 30866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:09:17.847905 30866 catalog_manager.cc:1347] Generated new cluster ID: e850aa356c054889a3aff709df98486b
I20260812 07:09:17.848054 30866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:09:17.873087 30866 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:09:17.873777 30866 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:09:17.886369 30866 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa4766585198468e9d5f6f26fb6a3d31: Generated new TSK 0
I20260812 07:09:17.887494 30866 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:09:17.903604 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 30771
W20260812 07:09:17.905599 30840 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41623
W20260812 07:09:17.906689 6388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 30771: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 07:09:17.906953 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 30771
2026-08-12T07:09:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2358 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:09:18 dist-test-slave-69rg krb5kdc[30880](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:09:18 dist-test-slave-69rg krb5kdc[30880](info): set up 2 sockets
Aug 12 07:09:18 dist-test-slave-69rg krb5kdc[30880](info): commencing operation
krb5kdc: starting...
W20260812 07:09:19.292327 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.341s user 0.000s sys 0.061s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:09:19 dist-test-slave-69rg krb5kdc[30880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:09:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:09:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 07:09:19.682790 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36849
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:09:20.062034 30896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:09:20.062419 30896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:09:20.062631 30896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:09:20.072131 30896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:09:20.072328 30896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:09:20.072435 30896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:09:20.072542 30896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:09:20.072629 30896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:09:20.084108 30896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:09:20.085971 30896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:09:20.087903 30896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:09:20.104323 30903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:20.104236 30905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:20.104300 30902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:09:20.105197 30896 server_base.cc:1034] running on GCE node
I20260812 07:09:20.106138 30896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:09:20.107793 30896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:09:20.109032 30896 hybrid_clock.cc:648] HybridClock initialized: now 1786518560108976 us; error 47 us; skew 500 ppm
I20260812 07:09:20.109422 30896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:09:20 dist-test-slave-69rg krb5kdc[30880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:09:20.113482 30896 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 07:09:20.115597 30896 webserver.cc:460] Webserver started at http://127.0.0.1:42889/ using document root <none> and password file <none>
I20260812 07:09:20.116400 30896 fs_manager.cc:359] Metadata directory not provided
I20260812 07:09:20.116562 30896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:09:20.116905 30896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:09:20.120360 30896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "15bb6d0dea18447d9a01bb94d856d329"
format_stamp: "Formatted at 2026-08-12 07:09:20 on dist-test-slave-69rg"
server_key: "18fb8cca788bbcba571bf23d523d50f2"
server_key_iv: "60e8c926cfa5ef82ff04af6deb3296fa"
server_key_version: "encryptionkey@0"
I20260812 07:09:20.121270 30896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "15bb6d0dea18447d9a01bb94d856d329"
format_stamp: "Formatted at 2026-08-12 07:09:20 on dist-test-slave-69rg"
server_key: "18fb8cca788bbcba571bf23d523d50f2"
server_key_iv: "60e8c926cfa5ef82ff04af6deb3296fa"
server_key_version: "encryptionkey@0"
I20260812 07:09:20.127936 30896 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260812 07:09:20.132507 30914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:09:20.134122 30896 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 07:09:20.134378 30896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "15bb6d0dea18447d9a01bb94d856d329"
format_stamp: "Formatted at 2026-08-12 07:09:20 on dist-test-slave-69rg"
server_key: "18fb8cca788bbcba571bf23d523d50f2"
server_key_iv: "60e8c926cfa5ef82ff04af6deb3296fa"
server_key_version: "encryptionkey@0"
I20260812 07:09:20.134620 30896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:09:20.148662 30896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:09:20.154470 30896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:09:20.154821 30896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:09:20.178750 30896 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36849
I20260812 07:09:20.179018 30967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36849 every 8 connection(s)
I20260812 07:09:20.180438 30896 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:09:20.182440 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 30896
I20260812 07:09:20.182662 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:09:20.183264 6388 external_mini_cluster.cc:1428] Setting key 32d1a6e052a196907d31d81778177ad8
I20260812 07:09:20.185320 30969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 07:09:20 dist-test-slave-69rg krb5kdc[30880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 07:09:20.202247 30969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15bb6d0dea18447d9a01bb94d856d329: Bootstrap starting.
I20260812 07:09:20.207000 30969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15bb6d0dea18447d9a01bb94d856d329: Neither blocks nor log segments found. Creating new log.
I20260812 07:09:20.208340 30969 log.cc:824] T 00000000000000000000000000000000 P 15bb6d0dea18447d9a01bb94d856d329: Log is configured to *not* fsync() on all Append() calls
I20260812 07:09:20.212592 30969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15bb6d0dea18447d9a01bb94d856d329: No bootstrap required, opened a new log
I20260812 07:09:20.213151 30972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:20.185724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:45489 (local address 127.0.0.1:36849)
0812 07:09:20.186895 (+ 1171us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:20.186909 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:20.186954 (+ 45us) server_negotiation.cc:407] Connection header received
0812 07:09:20.188914 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:20.188941 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:20.189446 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:20.190076 (+ 630us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:20.192218 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:20.193675 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:20.195667 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:20.196454 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:20.196610 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:20.202263 (+ 5653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:20.202280 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:20.202306 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:20.202365 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:20.206656 (+ 4291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:20.208052 (+ 1396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:20.208065 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:20.208076 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:20.208329 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:20.208992 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:20.209001 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:20.209011 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:20.210168 (+ 1157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:20.210552 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:20.211015 (+ 463us) server_negotiation.cc:299] Negotiation successful
0812 07:09:20.211416 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":912,"thread_start_us":595,"threads_started":1}
I20260812 07:09:20.218158 6388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:09:20.222338 30969 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15bb6d0dea18447d9a01bb94d856d329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15bb6d0dea18447d9a01bb94d856d329" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36849 } }
I20260812 07:09:20.222869 30969 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15bb6d0dea18447d9a01bb94d856d329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:09:20.223069 30969 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15bb6d0dea18447d9a01bb94d856d329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15bb6d0dea18447d9a01bb94d856d329, State: Initialized, Role: FOLLOWER
I20260812 07:09:20.223855 30969 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15bb6d0dea18447d9a01bb94d856d329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15bb6d0dea18447d9a01bb94d856d329" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36849 } }
I20260812 07:09:20.224177 30969 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15bb6d0dea18447d9a01bb94d856d329 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:09:20.224352 30969 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15bb6d0dea18447d9a01bb94d856d329 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:09:20.224665 30969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15bb6d0dea18447d9a01bb94d856d329 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:09:20.227782 30969 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15bb6d0dea18447d9a01bb94d856d329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15bb6d0dea18447d9a01bb94d856d329" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36849 } }
I20260812 07:09:20.228590 30969 leader_election.cc:302] T 00000000000000000000000000000000 P 15bb6d0dea18447d9a01bb94d856d329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15bb6d0dea18447d9a01bb94d856d329; no voters:
I20260812 07:09:20.229918 30969 leader_election.cc:288] T 00000000000000000000000000000000 P 15bb6d0dea18447d9a01bb94d856d329 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:09:20.230039 30980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15bb6d0dea18447d9a01bb94d856d329 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:09:20.231053 30980 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15bb6d0dea18447d9a01bb94d856d329 [term 1 LEADER]: Becoming Leader. State: Replica: 15bb6d0dea18447d9a01bb94d856d329, State: Running, Role: LEADER
I20260812 07:09:20.232103 30980 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15bb6d0dea18447d9a01bb94d856d329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15bb6d0dea18447d9a01bb94d856d329" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36849 } }
I20260812 07:09:20.234220 30969 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15bb6d0dea18447d9a01bb94d856d329 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:09:20.236311 30984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15bb6d0dea18447d9a01bb94d856d329 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15bb6d0dea18447d9a01bb94d856d329" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15bb6d0dea18447d9a01bb94d856d329" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36849 } } }
I20260812 07:09:20.236680 30984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15bb6d0dea18447d9a01bb94d856d329 [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:20.235610 30982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15bb6d0dea18447d9a01bb94d856d329 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15bb6d0dea18447d9a01bb94d856d329. Latest consensus state: current_term: 1 leader_uuid: "15bb6d0dea18447d9a01bb94d856d329" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15bb6d0dea18447d9a01bb94d856d329" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36849 } } }
I20260812 07:09:20.237222 30982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15bb6d0dea18447d9a01bb94d856d329 [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:20.248208 30972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:20.222759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:57205 (local address 127.0.0.1:36849)
0812 07:09:20.222986 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:20.222996 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:20.223553 (+ 557us) server_negotiation.cc:407] Connection header received
0812 07:09:20.223762 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:20.223766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:20.223845 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:20.223954 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:20.226068 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:20.226969 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:20.229117 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:20.230399 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:20.230571 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:20.234933 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:20.234945 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:20.234949 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:20.234996 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:20.240443 (+ 5447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:20.242271 (+ 1828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:20.242279 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:20.242287 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:20.242607 (+ 320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:20.243516 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:20.243526 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:20.243533 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:20.244164 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:20.244871 (+ 707us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:20.247285 (+ 2414us) server_negotiation.cc:299] Negotiation successful
0812 07:09:20.247598 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 07:09:20.250738 30987 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:09:20.257349 30987 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:09:20.267010 30987 catalog_manager.cc:1347] Generated new cluster ID: ad11b527844d402d8167236efd56c825
I20260812 07:09:20.267166 30987 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:09:20.281438 30987 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:09:20.283535 30987 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:09:20.297441 30987 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15bb6d0dea18447d9a01bb94d856d329: Generated new TSK 0
I20260812 07:09:20.298646 30987 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:09:20.314306 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 30896
I20260812 07:09:20.316373 30963 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:09:20.420504 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 30896
W20260812 07:09:20.448127 30878 connection.cc:535] client connection to 127.0.0.1:36849 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36849): Connection reset by peer (error 104)
2026-08-12T07:09:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2517 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:09:20 dist-test-slave-69rg krb5kdc[31004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:09:20 dist-test-slave-69rg krb5kdc[31004](info): set up 2 sockets
Aug 12 07:09:20 dist-test-slave-69rg krb5kdc[31004](info): commencing operation
krb5kdc: starting...
W20260812 07:09:21.700953 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.232s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:09:21 dist-test-slave-69rg krb5kdc[31004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:09:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:09:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 07:09:22.095290 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34685
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:09:22.478914 31020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:09:22.479341 31020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:09:22.479532 31020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:09:22.487192 31020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:09:22.487392 31020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:09:22.487504 31020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:09:22.487613 31020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:09:22.487699 31020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:09:22.498656 31020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:09:22.500095 31020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:09:22.501502 31020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:09:22.514528 31025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:22.514515 31026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:22.514555 31029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:09:22.515576 31020 server_base.cc:1034] running on GCE node
I20260812 07:09:22.516543 31020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:09:22.518175 31020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:09:22.519456 31020 hybrid_clock.cc:648] HybridClock initialized: now 1786518562519406 us; error 50 us; skew 500 ppm
I20260812 07:09:22.519886 31020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:09:22 dist-test-slave-69rg krb5kdc[31004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:09:22.523636 31020 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 07:09:22.525463 31020 webserver.cc:460] Webserver started at http://127.0.0.1:41775/ using document root <none> and password file <none>
I20260812 07:09:22.526366 31020 fs_manager.cc:359] Metadata directory not provided
I20260812 07:09:22.526525 31020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:09:22.526862 31020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:09:22.530104 31020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "df606956af2349f39985cf69c28175a4"
format_stamp: "Formatted at 2026-08-12 07:09:22 on dist-test-slave-69rg"
server_key: "9e7f6cf5412698a5a42fb0ed952fc5c9"
server_key_iv: "2e16039a5d3d7882f5cd767e94d58dfa"
server_key_version: "encryptionkey@0"
I20260812 07:09:22.530956 31020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "df606956af2349f39985cf69c28175a4"
format_stamp: "Formatted at 2026-08-12 07:09:22 on dist-test-slave-69rg"
server_key: "9e7f6cf5412698a5a42fb0ed952fc5c9"
server_key_iv: "2e16039a5d3d7882f5cd767e94d58dfa"
server_key_version: "encryptionkey@0"
I20260812 07:09:22.536484 31020 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 07:09:22.540328 31036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:09:22.542366 31020 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 07:09:22.542620 31020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "df606956af2349f39985cf69c28175a4"
format_stamp: "Formatted at 2026-08-12 07:09:22 on dist-test-slave-69rg"
server_key: "9e7f6cf5412698a5a42fb0ed952fc5c9"
server_key_iv: "2e16039a5d3d7882f5cd767e94d58dfa"
server_key_version: "encryptionkey@0"
I20260812 07:09:22.542872 31020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:09:22.563632 31020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:09:22.568336 31020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:09:22.568648 31020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:09:22.592027 31020 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34685
I20260812 07:09:22.592042 31092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34685 every 8 connection(s)
I20260812 07:09:22.593669 31020 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:09:22.597985 31093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:09:22.602304 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 31020
I20260812 07:09:22.602522 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:09:22.603046 6388 external_mini_cluster.cc:1428] Setting key b45546df6b0cb28f8e059ac7bf05efe3
I20260812 07:09:22.610030 31093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df606956af2349f39985cf69c28175a4: Bootstrap starting.
I20260812 07:09:22.614064 31093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df606956af2349f39985cf69c28175a4: Neither blocks nor log segments found. Creating new log.
I20260812 07:09:22.615386 31093 log.cc:824] T 00000000000000000000000000000000 P df606956af2349f39985cf69c28175a4: Log is configured to *not* fsync() on all Append() calls
I20260812 07:09:22.618964 31093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df606956af2349f39985cf69c28175a4: No bootstrap required, opened a new log
I20260812 07:09:22.619385 31096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:22.605188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:55617 (local address 127.0.0.1:34685)
0812 07:09:22.606094 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:22.606108 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:22.606219 (+ 111us) server_negotiation.cc:407] Connection header received
0812 07:09:22.608531 (+ 2312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:22.608559 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:22.609033 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:22.609423 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:22.610270 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:22.611786 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:22.613909 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:22.614803 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:22.614957 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:22.616578 (+ 1621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:22.616594 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:22.616618 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:09:22.617399 (+ 781us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:22.617466 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:09:22.617494 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:22.617629 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:22.618019 (+ 390us) server_negotiation.cc:299] Negotiation successful
0812 07:09:22.618300 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":405,"threads_started":1}
I20260812 07:09:22.623752 6388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:09:22.627247 31093 raft_consensus.cc:348] T 00000000000000000000000000000000 P df606956af2349f39985cf69c28175a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df606956af2349f39985cf69c28175a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34685 } }
I20260812 07:09:22.627660 31093 raft_consensus.cc:374] T 00000000000000000000000000000000 P df606956af2349f39985cf69c28175a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:09:22.627789 31093 raft_consensus.cc:729] T 00000000000000000000000000000000 P df606956af2349f39985cf69c28175a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df606956af2349f39985cf69c28175a4, State: Initialized, Role: FOLLOWER
I20260812 07:09:22.628398 31093 consensus_queue.cc:260] T 00000000000000000000000000000000 P df606956af2349f39985cf69c28175a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df606956af2349f39985cf69c28175a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34685 } }
I20260812 07:09:22.628805 31093 raft_consensus.cc:388] T 00000000000000000000000000000000 P df606956af2349f39985cf69c28175a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:09:22.629009 31093 raft_consensus.cc:482] T 00000000000000000000000000000000 P df606956af2349f39985cf69c28175a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:09:22.629314 31093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df606956af2349f39985cf69c28175a4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:09:22.631831 31093 raft_consensus.cc:504] T 00000000000000000000000000000000 P df606956af2349f39985cf69c28175a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df606956af2349f39985cf69c28175a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34685 } }
I20260812 07:09:22.632361 31093 leader_election.cc:302] T 00000000000000000000000000000000 P df606956af2349f39985cf69c28175a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df606956af2349f39985cf69c28175a4; no voters:
I20260812 07:09:22.633082 31093 leader_election.cc:288] T 00000000000000000000000000000000 P df606956af2349f39985cf69c28175a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:09:22.633200 31104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df606956af2349f39985cf69c28175a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:09:22.634732 31104 raft_consensus.cc:686] T 00000000000000000000000000000000 P df606956af2349f39985cf69c28175a4 [term 1 LEADER]: Becoming Leader. State: Replica: df606956af2349f39985cf69c28175a4, State: Running, Role: LEADER
I20260812 07:09:22.635469 31104 consensus_queue.cc:237] T 00000000000000000000000000000000 P df606956af2349f39985cf69c28175a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df606956af2349f39985cf69c28175a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34685 } }
I20260812 07:09:22.635824 31093 sys_catalog.cc:564] T 00000000000000000000000000000000 P df606956af2349f39985cf69c28175a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:09:22.636909 31096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:22.629012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:60467 (local address 127.0.0.1:34685)
0812 07:09:22.629275 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:22.629284 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:22.629717 (+ 433us) server_negotiation.cc:407] Connection header received
0812 07:09:22.629976 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:22.629982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:22.630074 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:22.630206 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:22.630793 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:22.631651 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:22.633388 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:22.634277 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:22.635050 (+ 773us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:22.636042 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:22.636049 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:22.636053 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:09:22.636074 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:22.636116 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:09:22.636134 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:22.636275 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:22.636504 (+ 229us) server_negotiation.cc:299] Negotiation successful
0812 07:09:22.636565 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 07:09:22.638943 31106 sys_catalog.cc:455] T 00000000000000000000000000000000 P df606956af2349f39985cf69c28175a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df606956af2349f39985cf69c28175a4. Latest consensus state: current_term: 1 leader_uuid: "df606956af2349f39985cf69c28175a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df606956af2349f39985cf69c28175a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34685 } } }
I20260812 07:09:22.639439 31106 sys_catalog.cc:458] T 00000000000000000000000000000000 P df606956af2349f39985cf69c28175a4 [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:22.640168 31105 sys_catalog.cc:455] T 00000000000000000000000000000000 P df606956af2349f39985cf69c28175a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df606956af2349f39985cf69c28175a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df606956af2349f39985cf69c28175a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34685 } } }
I20260812 07:09:22.640533 31105 sys_catalog.cc:458] T 00000000000000000000000000000000 P df606956af2349f39985cf69c28175a4 [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:22.647472 31109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:09:22.653298 31109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:09:22.663538 31109 catalog_manager.cc:1347] Generated new cluster ID: 530a4f4f0f824179bea5852b76377783
I20260812 07:09:22.663694 31109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:09:22.680456 31109 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:09:22.682503 31109 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:09:22.698906 31109 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df606956af2349f39985cf69c28175a4: Generated new TSK 0
I20260812 07:09:22.700199 31109 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:09:22.714974 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 31020
W20260812 07:09:22.716637 31088 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.162:55617
W20260812 07:09:22.717541 6388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 31020: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 07:09:22.717747 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 31020
W20260812 07:09:22.741250 31002 connection.cc:535] client connection to 127.0.0.1:34685 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34685): Connection reset by peer (error 104)
2026-08-12T07:09:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:09:22 dist-test-slave-69rg krb5kdc[31131](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:09:22 dist-test-slave-69rg krb5kdc[31131](info): set up 2 sockets
Aug 12 07:09:22 dist-test-slave-69rg krb5kdc[31131](info): commencing operation
krb5kdc: starting...
W20260812 07:09:24.005996 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.245s user 0.004s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:09:24 dist-test-slave-69rg krb5kdc[31131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:09:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:09:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 07:09:24.424598 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44243
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:09:24.849345 31146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:09:24.849704 31146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:09:24.849807 31146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:09:24.858693 31146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:09:24.858850 31146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:09:24.858901 31146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:09:24.858958 31146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:09:24.859006 31146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:09:24.872772 31146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:09:24.874385 31146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:09:24.875949 31146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:09:24.891494 31151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:24.891515 31152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:24.891530 31155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:09:24.893399 31146 server_base.cc:1034] running on GCE node
I20260812 07:09:24.894425 31146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:09:24.896070 31146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:09:24.897439 31146 hybrid_clock.cc:648] HybridClock initialized: now 1786518564897365 us; error 46 us; skew 500 ppm
I20260812 07:09:24.897965 31146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:09:24 dist-test-slave-69rg krb5kdc[31131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:09:24.910374 31146 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 07:09:24.912379 31146 webserver.cc:460] Webserver started at http://127.0.0.1:35581/ using document root <none> and password file <none>
I20260812 07:09:24.913209 31146 fs_manager.cc:359] Metadata directory not provided
I20260812 07:09:24.913396 31146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:09:24.913846 31146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:09:24.917383 31146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "d0c5613c40064b929e489ece3ecaf6cb"
format_stamp: "Formatted at 2026-08-12 07:09:24 on dist-test-slave-69rg"
server_key: "c8ce0cbaff3859bd31b6b5e57b5c323d"
server_key_iv: "bbbe5d8ae02eff3320bb7b4d13fad594"
server_key_version: "encryptionkey@0"
I20260812 07:09:24.918377 31146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "d0c5613c40064b929e489ece3ecaf6cb"
format_stamp: "Formatted at 2026-08-12 07:09:24 on dist-test-slave-69rg"
server_key: "c8ce0cbaff3859bd31b6b5e57b5c323d"
server_key_iv: "bbbe5d8ae02eff3320bb7b4d13fad594"
server_key_version: "encryptionkey@0"
I20260812 07:09:24.924888 31146 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 07:09:24.930559 31162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:09:24.932462 31146 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260812 07:09:24.932793 31146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "d0c5613c40064b929e489ece3ecaf6cb"
format_stamp: "Formatted at 2026-08-12 07:09:24 on dist-test-slave-69rg"
server_key: "c8ce0cbaff3859bd31b6b5e57b5c323d"
server_key_iv: "bbbe5d8ae02eff3320bb7b4d13fad594"
server_key_version: "encryptionkey@0"
I20260812 07:09:24.933046 31146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:09:24.940124 31146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:09:24.946098 31146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:09:24.946524 31146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:09:24.970016 31146 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44243
I20260812 07:09:24.970019 31222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44243 every 8 connection(s)
I20260812 07:09:24.971565 31146 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:09:24.976917 31223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:09:24.977615 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 31146
I20260812 07:09:24.978042 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:09:24.978966 6388 external_mini_cluster.cc:1428] Setting key e2e42690d51273971b9c9fcf51761817
I20260812 07:09:24.990970 31226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:24.981701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:60123 (local address 127.0.0.1:44243)
0812 07:09:24.982935 (+ 1234us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:24.982956 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:24.983031 (+ 75us) server_negotiation.cc:407] Connection header received
0812 07:09:24.985774 (+ 2743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:24.985836 (+ 62us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:24.986441 (+ 605us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:24.987010 (+ 569us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:24.988199 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:24.988208 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:24.988227 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:09:24.988250 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:24.988336 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:09:24.988373 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:24.988749 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:24.988929 (+ 180us) server_negotiation.cc:299] Negotiation successful
0812 07:09:24.989293 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":906,"thread_start_us":542,"threads_started":1}
I20260812 07:09:24.996562 31223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0c5613c40064b929e489ece3ecaf6cb: Bootstrap starting.
I20260812 07:09:24.996836 6388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:09:25.000969 31223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0c5613c40064b929e489ece3ecaf6cb: Neither blocks nor log segments found. Creating new log.
I20260812 07:09:25.002700 31223 log.cc:824] T 00000000000000000000000000000000 P d0c5613c40064b929e489ece3ecaf6cb: Log is configured to *not* fsync() on all Append() calls
I20260812 07:09:25.006587 31223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0c5613c40064b929e489ece3ecaf6cb: No bootstrap required, opened a new log
I20260812 07:09:25.006997 31226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:25.003371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:36241 (local address 127.0.0.1:44243)
0812 07:09:25.003567 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:25.003576 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:25.004463 (+ 887us) server_negotiation.cc:407] Connection header received
0812 07:09:25.004765 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:25.004771 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:25.004830 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:25.004952 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:25.006201 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:25.006206 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:25.006209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:09:25.006228 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:25.006273 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:09:25.006291 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:25.006431 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:25.006582 (+ 151us) server_negotiation.cc:299] Negotiation successful
0812 07:09:25.006663 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 07:09:25.016227 31223 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0c5613c40064b929e489ece3ecaf6cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c5613c40064b929e489ece3ecaf6cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44243 } }
I20260812 07:09:25.016795 31223 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0c5613c40064b929e489ece3ecaf6cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:09:25.016920 31223 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0c5613c40064b929e489ece3ecaf6cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0c5613c40064b929e489ece3ecaf6cb, State: Initialized, Role: FOLLOWER
I20260812 07:09:25.018035 31223 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0c5613c40064b929e489ece3ecaf6cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c5613c40064b929e489ece3ecaf6cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44243 } }
I20260812 07:09:25.018726 31223 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0c5613c40064b929e489ece3ecaf6cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:09:25.018949 31223 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0c5613c40064b929e489ece3ecaf6cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:09:25.019165 31223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0c5613c40064b929e489ece3ecaf6cb [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:09:25.022183 31223 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0c5613c40064b929e489ece3ecaf6cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c5613c40064b929e489ece3ecaf6cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44243 } }
I20260812 07:09:25.022910 31223 leader_election.cc:302] T 00000000000000000000000000000000 P d0c5613c40064b929e489ece3ecaf6cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0c5613c40064b929e489ece3ecaf6cb; no voters:
I20260812 07:09:25.023773 31223 leader_election.cc:288] T 00000000000000000000000000000000 P d0c5613c40064b929e489ece3ecaf6cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:09:25.023957 31235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0c5613c40064b929e489ece3ecaf6cb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:09:25.024904 31235 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0c5613c40064b929e489ece3ecaf6cb [term 1 LEADER]: Becoming Leader. State: Replica: d0c5613c40064b929e489ece3ecaf6cb, State: Running, Role: LEADER
I20260812 07:09:25.025647 31235 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0c5613c40064b929e489ece3ecaf6cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c5613c40064b929e489ece3ecaf6cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44243 } }
I20260812 07:09:25.026569 31223 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0c5613c40064b929e489ece3ecaf6cb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:09:25.029029 31237 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0c5613c40064b929e489ece3ecaf6cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0c5613c40064b929e489ece3ecaf6cb. Latest consensus state: current_term: 1 leader_uuid: "d0c5613c40064b929e489ece3ecaf6cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c5613c40064b929e489ece3ecaf6cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44243 } } }
I20260812 07:09:25.030208 31237 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0c5613c40064b929e489ece3ecaf6cb [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:25.029649 31236 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0c5613c40064b929e489ece3ecaf6cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0c5613c40064b929e489ece3ecaf6cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c5613c40064b929e489ece3ecaf6cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44243 } } }
I20260812 07:09:25.030712 31236 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0c5613c40064b929e489ece3ecaf6cb [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:25.033309 31244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:09:25.039417 31244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:09:25.049309 31244 catalog_manager.cc:1347] Generated new cluster ID: 7a2e58aaf503474294cdd27fca9dae45
I20260812 07:09:25.049459 31244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:09:25.073796 31244 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:09:25.074453 31244 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:09:25.097460 31244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0c5613c40064b929e489ece3ecaf6cb: Generated new TSK 0
I20260812 07:09:25.098989 31244 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:09:25.105798 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 31146
W20260812 07:09:25.107547 31217 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.162:60123
W20260812 07:09:25.108613 6388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 31146: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 07:09:25.108816 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 31146
2026-08-12T07:09:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2391 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:09:25 dist-test-slave-69rg krb5kdc[31258](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:09:25 dist-test-slave-69rg krb5kdc[31258](info): set up 2 sockets
Aug 12 07:09:25 dist-test-slave-69rg krb5kdc[31258](info): commencing operation
krb5kdc: starting...
W20260812 07:09:26.490101 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.336s user 0.005s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:09:26 dist-test-slave-69rg krb5kdc[31258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:09:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:09:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 07:09:26.912674 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34155
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:09:27.325939 31274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:09:27.326289 31274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:09:27.326453 31274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:09:27.334085 31274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:09:27.334225 31274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:09:27.334275 31274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:09:27.334370 31274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:09:27.334438 31274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:09:27.345530 31274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34155
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:09:27.347559 31274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:09:27.349227 31274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:09:27.364373 31279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:27.364555 31282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:27.364605 31280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:09:27.365754 31274 server_base.cc:1034] running on GCE node
I20260812 07:09:27.366845 31274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:09:27.368472 31274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:09:27.369710 31274 hybrid_clock.cc:648] HybridClock initialized: now 1786518567369656 us; error 38 us; skew 500 ppm
I20260812 07:09:27.370162 31274 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:09:27 dist-test-slave-69rg krb5kdc[31258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:09:27.374239 31274 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 07:09:27.376474 31274 webserver.cc:460] Webserver started at http://127.0.0.1:42781/ using document root <none> and password file <none>
I20260812 07:09:27.377336 31274 fs_manager.cc:359] Metadata directory not provided
I20260812 07:09:27.377467 31274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:09:27.377789 31274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:09:27.381664 31274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "d06cd51095d94de596785cb41573d047"
format_stamp: "Formatted at 2026-08-12 07:09:27 on dist-test-slave-69rg"
server_key: "9693ae073694941bb8a17328fe437265"
server_key_iv: "7f18d17f0e0a4327bd307f9de87ebe78"
server_key_version: "encryptionkey@0"
I20260812 07:09:27.383015 31274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "d06cd51095d94de596785cb41573d047"
format_stamp: "Formatted at 2026-08-12 07:09:27 on dist-test-slave-69rg"
server_key: "9693ae073694941bb8a17328fe437265"
server_key_iv: "7f18d17f0e0a4327bd307f9de87ebe78"
server_key_version: "encryptionkey@0"
I20260812 07:09:27.390429 31274 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260812 07:09:27.395222 31289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:09:27.397892 31274 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260812 07:09:27.398312 31274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "d06cd51095d94de596785cb41573d047"
format_stamp: "Formatted at 2026-08-12 07:09:27 on dist-test-slave-69rg"
server_key: "9693ae073694941bb8a17328fe437265"
server_key_iv: "7f18d17f0e0a4327bd307f9de87ebe78"
server_key_version: "encryptionkey@0"
I20260812 07:09:27.398618 31274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:09:27.423802 31274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:09:27.429467 31274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:09:27.430240 31274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:09:27.455972 31274 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34155
I20260812 07:09:27.456001 31344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34155 every 8 connection(s)
I20260812 07:09:27.457669 31274 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:09:27.462963 31345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:09:27.466168 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 31274
I20260812 07:09:27.466394 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:09:27.466857 6388 external_mini_cluster.cc:1428] Setting key bcb9842d1cbebe31928b5902d469584f
I20260812 07:09:27.478351 31345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d06cd51095d94de596785cb41573d047: Bootstrap starting.
I20260812 07:09:27.483523 31345 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d06cd51095d94de596785cb41573d047: Neither blocks nor log segments found. Creating new log.
I20260812 07:09:27.485131 31345 log.cc:824] T 00000000000000000000000000000000 P d06cd51095d94de596785cb41573d047: Log is configured to *not* fsync() on all Append() calls
Aug 12 07:09:27 dist-test-slave-69rg krb5kdc[31258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 07:09:27.489517 31345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d06cd51095d94de596785cb41573d047: No bootstrap required, opened a new log
I20260812 07:09:27.498940 31348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:27.469324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:35961 (local address 127.0.0.1:34155)
0812 07:09:27.470335 (+ 1011us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:27.470349 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:27.470393 (+ 44us) server_negotiation.cc:407] Connection header received
0812 07:09:27.473063 (+ 2670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:27.473098 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:27.473961 (+ 863us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:27.474746 (+ 785us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:27.476846 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:27.478657 (+ 1811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:27.481136 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:27.482304 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:27.482492 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:27.488545 (+ 6053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:27.488566 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:27.488594 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:27.488645 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:27.493269 (+ 4624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:27.494550 (+ 1281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:27.494559 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:27.494567 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:27.494778 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:27.495391 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:27.495396 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:27.495398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:27.496092 (+ 694us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:27.496517 (+ 425us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:27.497179 (+ 662us) server_negotiation.cc:299] Negotiation successful
0812 07:09:27.497585 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":466,"threads_started":1}
I20260812 07:09:27.500185 31345 raft_consensus.cc:348] T 00000000000000000000000000000000 P d06cd51095d94de596785cb41573d047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d06cd51095d94de596785cb41573d047" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34155 } }
I20260812 07:09:27.500859 31345 raft_consensus.cc:374] T 00000000000000000000000000000000 P d06cd51095d94de596785cb41573d047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:09:27.500989 31345 raft_consensus.cc:729] T 00000000000000000000000000000000 P d06cd51095d94de596785cb41573d047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d06cd51095d94de596785cb41573d047, State: Initialized, Role: FOLLOWER
I20260812 07:09:27.501667 31345 consensus_queue.cc:260] T 00000000000000000000000000000000 P d06cd51095d94de596785cb41573d047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d06cd51095d94de596785cb41573d047" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34155 } }
I20260812 07:09:27.502020 31345 raft_consensus.cc:388] T 00000000000000000000000000000000 P d06cd51095d94de596785cb41573d047 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:09:27.502202 31345 raft_consensus.cc:482] T 00000000000000000000000000000000 P d06cd51095d94de596785cb41573d047 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:09:27.502393 31345 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d06cd51095d94de596785cb41573d047 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:09:27.505298 31345 raft_consensus.cc:504] T 00000000000000000000000000000000 P d06cd51095d94de596785cb41573d047 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d06cd51095d94de596785cb41573d047" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34155 } }
I20260812 07:09:27.505545 6388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:09:27.505997 31345 leader_election.cc:302] T 00000000000000000000000000000000 P d06cd51095d94de596785cb41573d047 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d06cd51095d94de596785cb41573d047; no voters:
I20260812 07:09:27.506769 31345 leader_election.cc:288] T 00000000000000000000000000000000 P d06cd51095d94de596785cb41573d047 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:09:27.506989 31350 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d06cd51095d94de596785cb41573d047 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:09:27.509852 31345 sys_catalog.cc:564] T 00000000000000000000000000000000 P d06cd51095d94de596785cb41573d047 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:09:27.514514 31350 raft_consensus.cc:686] T 00000000000000000000000000000000 P d06cd51095d94de596785cb41573d047 [term 1 LEADER]: Becoming Leader. State: Replica: d06cd51095d94de596785cb41573d047, State: Running, Role: LEADER
I20260812 07:09:27.515476 31350 consensus_queue.cc:237] T 00000000000000000000000000000000 P d06cd51095d94de596785cb41573d047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d06cd51095d94de596785cb41573d047" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34155 } }
I20260812 07:09:27.519382 31351 sys_catalog.cc:455] T 00000000000000000000000000000000 P d06cd51095d94de596785cb41573d047 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d06cd51095d94de596785cb41573d047" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d06cd51095d94de596785cb41573d047" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34155 } } }
I20260812 07:09:27.520237 31351 sys_catalog.cc:458] T 00000000000000000000000000000000 P d06cd51095d94de596785cb41573d047 [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:27.520927 31352 sys_catalog.cc:455] T 00000000000000000000000000000000 P d06cd51095d94de596785cb41573d047 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d06cd51095d94de596785cb41573d047. Latest consensus state: current_term: 1 leader_uuid: "d06cd51095d94de596785cb41573d047" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d06cd51095d94de596785cb41573d047" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34155 } } }
I20260812 07:09:27.521283 31352 sys_catalog.cc:458] T 00000000000000000000000000000000 P d06cd51095d94de596785cb41573d047 [sys.catalog]: This master's current role is: LEADER
W20260812 07:09:27.537964 31373 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d06cd51095d94de596785cb41573d047: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 07:09:27.538177 31373 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 07:09:27.538338 31374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:09:27.540997 31374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:09:27.547166 31348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:27.521490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:49591 (local address 127.0.0.1:34155)
0812 07:09:27.521742 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:27.521751 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:27.530903 (+ 9152us) server_negotiation.cc:407] Connection header received
0812 07:09:27.531169 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:27.531180 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:27.531334 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:27.531518 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:27.534219 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:27.535065 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:27.536603 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:27.537093 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:27.537279 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:27.540311 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:27.540317 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:27.540320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:27.540362 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:27.543736 (+ 3374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:27.544707 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:27.544712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:27.544714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:27.544853 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:27.545382 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:27.545386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:27.545388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:27.545753 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:27.546097 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:27.546554 (+ 457us) server_negotiation.cc:299] Negotiation successful
0812 07:09:27.546753 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 07:09:27.552663 31374 catalog_manager.cc:1347] Generated new cluster ID: 903caae4b912456a9fb10c11fd0c47f0
I20260812 07:09:27.552838 31374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:09:27.570355 31374 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:09:27.572597 31374 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:09:27.580631 31374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d06cd51095d94de596785cb41573d047: Generated new TSK 0
I20260812 07:09:27.581990 31374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:09:27.596951 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 31274
I20260812 07:09:27.598630 31339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:09:27.697504 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 31274
W20260812 07:09:27.723671 31256 connection.cc:535] client connection to 127.0.0.1:34155 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34155): Connection reset by peer (error 104)
2026-08-12T07:09:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2598 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:09:27 dist-test-slave-69rg krb5kdc[31382](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:09:27 dist-test-slave-69rg krb5kdc[31382](info): set up 2 sockets
Aug 12 07:09:27 dist-test-slave-69rg krb5kdc[31382](info): commencing operation
krb5kdc: starting...
W20260812 07:09:29.013986 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.262s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:09:29 dist-test-slave-69rg krb5kdc[31382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:09:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:09:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 07:09:29.378887 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41165
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:09:29.750548 31397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:09:29.750886 31397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:09:29.751037 31397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:09:29.758821 31397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:09:29.758960 31397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:09:29.759061 31397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:09:29.759164 31397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:09:29.759264 31397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:09:29.770443 31397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41165
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:09:29.772017 31397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:09:29.773571 31397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:09:29.787817 31403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:29.787894 31405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:29.787853 31402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:09:29.788523 31397 server_base.cc:1034] running on GCE node
I20260812 07:09:29.789505 31397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:09:29.791190 31397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:09:29.792480 31397 hybrid_clock.cc:648] HybridClock initialized: now 1786518569792422 us; error 42 us; skew 500 ppm
I20260812 07:09:29.792910 31397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:09:29 dist-test-slave-69rg krb5kdc[31382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:09:29.796468 31397 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 07:09:29.798180 31397 webserver.cc:460] Webserver started at http://127.0.0.1:33599/ using document root <none> and password file <none>
I20260812 07:09:29.798918 31397 fs_manager.cc:359] Metadata directory not provided
I20260812 07:09:29.799073 31397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:09:29.799410 31397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:09:29.802641 31397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "106f8e44073f46a884c4ad8171a14e17"
format_stamp: "Formatted at 2026-08-12 07:09:29 on dist-test-slave-69rg"
server_key: "cf89d3e8e1215500dd0787739e285026"
server_key_iv: "fef3042cfa0e614fa3fd0a78eba07026"
server_key_version: "encryptionkey@0"
I20260812 07:09:29.803491 31397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "106f8e44073f46a884c4ad8171a14e17"
format_stamp: "Formatted at 2026-08-12 07:09:29 on dist-test-slave-69rg"
server_key: "cf89d3e8e1215500dd0787739e285026"
server_key_iv: "fef3042cfa0e614fa3fd0a78eba07026"
server_key_version: "encryptionkey@0"
I20260812 07:09:29.809141 31397 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 07:09:29.813687 31413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:09:29.815358 31397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:09:29.815575 31397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "106f8e44073f46a884c4ad8171a14e17"
format_stamp: "Formatted at 2026-08-12 07:09:29 on dist-test-slave-69rg"
server_key: "cf89d3e8e1215500dd0787739e285026"
server_key_iv: "fef3042cfa0e614fa3fd0a78eba07026"
server_key_version: "encryptionkey@0"
I20260812 07:09:29.815819 31397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:09:29.837815 31397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:09:29.843786 31397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:09:29.844180 31397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:09:29.866586 31397 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41165
I20260812 07:09:29.866609 31469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41165 every 8 connection(s)
I20260812 07:09:29.868351 31397 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:09:29.872430 31470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:09:29.876365 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 31397
I20260812 07:09:29.876571 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:09:29.877146 6388 external_mini_cluster.cc:1428] Setting key e5a3f9c2cb0b7f2af72dad59b4027a0c
I20260812 07:09:29.885716 31470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 106f8e44073f46a884c4ad8171a14e17: Bootstrap starting.
I20260812 07:09:29.889011 31470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 106f8e44073f46a884c4ad8171a14e17: Neither blocks nor log segments found. Creating new log.
I20260812 07:09:29.890283 31470 log.cc:824] T 00000000000000000000000000000000 P 106f8e44073f46a884c4ad8171a14e17: Log is configured to *not* fsync() on all Append() calls
I20260812 07:09:29.891685 31474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:29.879347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:42299 (local address 127.0.0.1:41165)
0812 07:09:29.880182 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:29.880197 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:29.880246 (+ 49us) server_negotiation.cc:407] Connection header received
0812 07:09:29.882148 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:29.882174 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:29.882639 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:29.883070 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:29.883700 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:29.885030 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:29.886607 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:29.887326 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:29.887545 (+ 219us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:29.888850 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:29.888858 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:29.888874 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:09:29.889537 (+ 663us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:29.889617 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:09:29.889652 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:29.889826 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:29.890169 (+ 343us) server_negotiation.cc:299] Negotiation successful
0812 07:09:29.890350 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":358,"threads_started":1}
I20260812 07:09:29.893452 31470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 106f8e44073f46a884c4ad8171a14e17: No bootstrap required, opened a new log
I20260812 07:09:29.895737 6388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:09:29.902529 31470 raft_consensus.cc:348] T 00000000000000000000000000000000 P 106f8e44073f46a884c4ad8171a14e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106f8e44073f46a884c4ad8171a14e17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41165 } }
I20260812 07:09:29.903101 31470 raft_consensus.cc:374] T 00000000000000000000000000000000 P 106f8e44073f46a884c4ad8171a14e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:09:29.903235 31470 raft_consensus.cc:729] T 00000000000000000000000000000000 P 106f8e44073f46a884c4ad8171a14e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 106f8e44073f46a884c4ad8171a14e17, State: Initialized, Role: FOLLOWER
I20260812 07:09:29.903967 31470 consensus_queue.cc:260] T 00000000000000000000000000000000 P 106f8e44073f46a884c4ad8171a14e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106f8e44073f46a884c4ad8171a14e17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41165 } }
I20260812 07:09:29.904273 31470 raft_consensus.cc:388] T 00000000000000000000000000000000 P 106f8e44073f46a884c4ad8171a14e17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:09:29.904441 31470 raft_consensus.cc:482] T 00000000000000000000000000000000 P 106f8e44073f46a884c4ad8171a14e17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:09:29.904620 31470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 106f8e44073f46a884c4ad8171a14e17 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:09:29.907550 31470 raft_consensus.cc:504] T 00000000000000000000000000000000 P 106f8e44073f46a884c4ad8171a14e17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106f8e44073f46a884c4ad8171a14e17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41165 } }
I20260812 07:09:29.908116 31470 leader_election.cc:302] T 00000000000000000000000000000000 P 106f8e44073f46a884c4ad8171a14e17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 106f8e44073f46a884c4ad8171a14e17; no voters:
I20260812 07:09:29.908833 31470 leader_election.cc:288] T 00000000000000000000000000000000 P 106f8e44073f46a884c4ad8171a14e17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:09:29.909063 31481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 106f8e44073f46a884c4ad8171a14e17 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:09:29.909962 31481 raft_consensus.cc:686] T 00000000000000000000000000000000 P 106f8e44073f46a884c4ad8171a14e17 [term 1 LEADER]: Becoming Leader. State: Replica: 106f8e44073f46a884c4ad8171a14e17, State: Running, Role: LEADER
I20260812 07:09:29.910657 31481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 106f8e44073f46a884c4ad8171a14e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106f8e44073f46a884c4ad8171a14e17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41165 } }
I20260812 07:09:29.911255 31470 sys_catalog.cc:564] T 00000000000000000000000000000000 P 106f8e44073f46a884c4ad8171a14e17 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:09:29.914106 31474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:29.901008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:46293 (local address 127.0.0.1:41165)
0812 07:09:29.901262 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:29.901268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:29.901741 (+ 473us) server_negotiation.cc:407] Connection header received
0812 07:09:29.901986 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:29.901991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:29.902085 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:29.902211 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:29.902830 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:29.904012 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:29.905984 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:29.906752 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:29.906879 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:29.912470 (+ 5591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:29.912482 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:29.912491 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:09:29.912524 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:29.912628 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:09:29.912663 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:29.912841 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:29.913452 (+ 611us) server_negotiation.cc:299] Negotiation successful
0812 07:09:29.913530 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260812 07:09:29.913959 31483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 106f8e44073f46a884c4ad8171a14e17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 106f8e44073f46a884c4ad8171a14e17. Latest consensus state: current_term: 1 leader_uuid: "106f8e44073f46a884c4ad8171a14e17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106f8e44073f46a884c4ad8171a14e17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41165 } } }
I20260812 07:09:29.914434 31483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 106f8e44073f46a884c4ad8171a14e17 [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:29.915452 31482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 106f8e44073f46a884c4ad8171a14e17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "106f8e44073f46a884c4ad8171a14e17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106f8e44073f46a884c4ad8171a14e17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41165 } } }
I20260812 07:09:29.915771 31482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 106f8e44073f46a884c4ad8171a14e17 [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:29.917807 31489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:09:29.924886 31489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:09:29.936512 31489 catalog_manager.cc:1347] Generated new cluster ID: 1994faf2ba714df488d91f8b9fd72df4
I20260812 07:09:29.936662 31489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:09:29.960920 31489 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:09:29.962393 31489 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:09:29.970904 31489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 106f8e44073f46a884c4ad8171a14e17: Generated new TSK 0
I20260812 07:09:29.972102 31489 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:09:29.983469 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 31397
W20260812 07:09:29.985407 31465 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.162:42299
W20260812 07:09:29.986321 6388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 31397: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 07:09:29.986490 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 31397
W20260812 07:09:30.010651 31380 connection.cc:535] client connection to 127.0.0.1:41165 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41165): Connection reset by peer (error 104)
2026-08-12T07:09:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2279 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:09:30 dist-test-slave-69rg krb5kdc[31506](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:09:30 dist-test-slave-69rg krb5kdc[31506](info): set up 2 sockets
Aug 12 07:09:30 dist-test-slave-69rg krb5kdc[31506](info): commencing operation
krb5kdc: starting...
W20260812 07:09:31.270557 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.238s user 0.002s sys 0.053s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:09:31 dist-test-slave-69rg krb5kdc[31506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:09:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:09:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 07:09:31.659247 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43461
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:09:32.044289 31523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:09:32.044634 31523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:09:32.044734 31523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:09:32.052496 31523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:09:32.052649 31523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:09:32.052708 31523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:09:32.052772 31523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:09:32.052820 31523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:09:32.066829 31523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43461
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518208686283-6388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:09:32.068192 31523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:09:32.069547 31523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:09:32.082697 31529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:32.082697 31530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:32.082713 31532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:09:32.084574 31523 server_base.cc:1034] running on GCE node
I20260812 07:09:32.085541 31523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:09:32.087199 31523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:09:32.088469 31523 hybrid_clock.cc:648] HybridClock initialized: now 1786518572088419 us; error 37 us; skew 500 ppm
I20260812 07:09:32.088876 31523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:09:32 dist-test-slave-69rg krb5kdc[31506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:09:32.092502 31523 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 07:09:32.094355 31523 webserver.cc:460] Webserver started at http://127.0.0.1:40739/ using document root <none> and password file <none>
I20260812 07:09:32.095096 31523 fs_manager.cc:359] Metadata directory not provided
I20260812 07:09:32.095256 31523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:09:32.095604 31523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:09:32.099002 31523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "40d3ff49ce7e477191a7e05eb924d11e"
format_stamp: "Formatted at 2026-08-12 07:09:32 on dist-test-slave-69rg"
server_key: "bc1a5f06a5956708e8d9731a6a2fe1c1"
server_key_iv: "048d4d7f36c09ae659b366e5491bbb34"
server_key_version: "encryptionkey@0"
I20260812 07:09:32.099870 31523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "40d3ff49ce7e477191a7e05eb924d11e"
format_stamp: "Formatted at 2026-08-12 07:09:32 on dist-test-slave-69rg"
server_key: "bc1a5f06a5956708e8d9731a6a2fe1c1"
server_key_iv: "048d4d7f36c09ae659b366e5491bbb34"
server_key_version: "encryptionkey@0"
I20260812 07:09:32.105963 31523 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260812 07:09:32.109985 31542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:09:32.111773 31523 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 07:09:32.112035 31523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "40d3ff49ce7e477191a7e05eb924d11e"
format_stamp: "Formatted at 2026-08-12 07:09:32 on dist-test-slave-69rg"
server_key: "bc1a5f06a5956708e8d9731a6a2fe1c1"
server_key_iv: "048d4d7f36c09ae659b366e5491bbb34"
server_key_version: "encryptionkey@0"
I20260812 07:09:32.112300 31523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:09:32.119489 31523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:09:32.125263 31523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:09:32.125656 31523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:09:32.148789 31523 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43461
I20260812 07:09:32.148854 31600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43461 every 8 connection(s)
I20260812 07:09:32.150609 31523 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:09:32.156154 31601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:09:32.158365 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 31523
I20260812 07:09:32.158583 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:09:32.159211 6388 external_mini_cluster.cc:1428] Setting key 9630752c8fbf4d22c2f359304005cbeb
I20260812 07:09:32.169696 31604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:32.161157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:38277 (local address 127.0.0.1:43461)
0812 07:09:32.162162 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:32.162179 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:32.162231 (+ 52us) server_negotiation.cc:407] Connection header received
0812 07:09:32.164812 (+ 2581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:32.164848 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:32.165389 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:32.166025 (+ 636us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:32.167207 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:32.167216 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:32.167234 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:09:32.167255 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:32.167340 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:09:32.167370 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:32.167615 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:32.167881 (+ 266us) server_negotiation.cc:299] Negotiation successful
0812 07:09:32.168152 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":392,"threads_started":1}
I20260812 07:09:32.174680 6388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:09:32.174784 31601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40d3ff49ce7e477191a7e05eb924d11e: Bootstrap starting.
I20260812 07:09:32.178469 31601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40d3ff49ce7e477191a7e05eb924d11e: Neither blocks nor log segments found. Creating new log.
I20260812 07:09:32.179957 31601 log.cc:824] T 00000000000000000000000000000000 P 40d3ff49ce7e477191a7e05eb924d11e: Log is configured to *not* fsync() on all Append() calls
I20260812 07:09:32.182863 31604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:32.179854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:37733 (local address 127.0.0.1:43461)
0812 07:09:32.180080 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:32.180089 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:32.180548 (+ 459us) server_negotiation.cc:407] Connection header received
0812 07:09:32.180797 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:32.180803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:32.180869 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:32.181005 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:32.182117 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:32.182124 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:32.182130 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:09:32.182150 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:32.182200 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:09:32.182218 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:32.182360 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:32.182475 (+ 115us) server_negotiation.cc:299] Negotiation successful
0812 07:09:32.182552 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 07:09:32.183876 31601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40d3ff49ce7e477191a7e05eb924d11e: No bootstrap required, opened a new log
I20260812 07:09:32.192694 31601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40d3ff49ce7e477191a7e05eb924d11e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d3ff49ce7e477191a7e05eb924d11e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43461 } }
I20260812 07:09:32.193159 31601 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40d3ff49ce7e477191a7e05eb924d11e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:09:32.193289 31601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40d3ff49ce7e477191a7e05eb924d11e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40d3ff49ce7e477191a7e05eb924d11e, State: Initialized, Role: FOLLOWER
I20260812 07:09:32.194037 31601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40d3ff49ce7e477191a7e05eb924d11e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d3ff49ce7e477191a7e05eb924d11e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43461 } }
I20260812 07:09:32.194383 31601 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40d3ff49ce7e477191a7e05eb924d11e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:09:32.194547 31601 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40d3ff49ce7e477191a7e05eb924d11e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:09:32.194741 31601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40d3ff49ce7e477191a7e05eb924d11e [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:09:32.197077 31601 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40d3ff49ce7e477191a7e05eb924d11e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d3ff49ce7e477191a7e05eb924d11e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43461 } }
I20260812 07:09:32.197685 31601 leader_election.cc:302] T 00000000000000000000000000000000 P 40d3ff49ce7e477191a7e05eb924d11e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40d3ff49ce7e477191a7e05eb924d11e; no voters:
I20260812 07:09:32.198493 31601 leader_election.cc:288] T 00000000000000000000000000000000 P 40d3ff49ce7e477191a7e05eb924d11e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:09:32.198642 31612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40d3ff49ce7e477191a7e05eb924d11e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:09:32.199365 31612 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40d3ff49ce7e477191a7e05eb924d11e [term 1 LEADER]: Becoming Leader. State: Replica: 40d3ff49ce7e477191a7e05eb924d11e, State: Running, Role: LEADER
I20260812 07:09:32.200017 31612 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40d3ff49ce7e477191a7e05eb924d11e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d3ff49ce7e477191a7e05eb924d11e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43461 } }
I20260812 07:09:32.201015 31601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40d3ff49ce7e477191a7e05eb924d11e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:09:32.203500 31613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40d3ff49ce7e477191a7e05eb924d11e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40d3ff49ce7e477191a7e05eb924d11e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d3ff49ce7e477191a7e05eb924d11e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43461 } } }
I20260812 07:09:32.203483 31614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40d3ff49ce7e477191a7e05eb924d11e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40d3ff49ce7e477191a7e05eb924d11e. Latest consensus state: current_term: 1 leader_uuid: "40d3ff49ce7e477191a7e05eb924d11e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d3ff49ce7e477191a7e05eb924d11e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43461 } } }
I20260812 07:09:32.204000 31613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40d3ff49ce7e477191a7e05eb924d11e [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:32.204011 31614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40d3ff49ce7e477191a7e05eb924d11e [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:32.206485 31619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:09:32.212509 31619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:09:32.222880 31619 catalog_manager.cc:1347] Generated new cluster ID: 48d2775a80e34974a2c7a93f3f0906d1
I20260812 07:09:32.223023 31619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:09:32.242547 31619 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:09:32.243301 31619 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:09:32.255565 31619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40d3ff49ce7e477191a7e05eb924d11e: Generated new TSK 0
I20260812 07:09:32.256877 31619 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:09:32.270804 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 31523
W20260812 07:09:32.272612 31596 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.162:38277
W20260812 07:09:32.273757 6388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 31523: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 07:09:32.273989 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 31523
2026-08-12T07:09:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2285 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:09:32 dist-test-slave-69rg krb5kdc[31637](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:09:32 dist-test-slave-69rg krb5kdc[31637](info): set up 2 sockets
Aug 12 07:09:32 dist-test-slave-69rg krb5kdc[31637](info): commencing operation
krb5kdc: starting...
W20260812 07:09:33.532644 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.216s user 0.000s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:09:33 dist-test-slave-69rg krb5kdc[31637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:09:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:09:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
I20260812 07:09:33.903340 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:33697
--webserver_interface=127.6.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:43355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:33697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:09:34.286325 31653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:09:34.286715 31653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:09:34.286892 31653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:09:34.294690 31653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:09:34.294867 31653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:09:34.294979 31653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:09:34.295094 31653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:09:34.295185 31653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:09:34.305713 31653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:43355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:33697
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:33697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:09:34.307284 31653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:09:34.308722 31653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:09:34.322559 31659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:34.322629 31662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:34.322667 31660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:09:34.324563 31653 server_base.cc:1034] running on GCE node
I20260812 07:09:34.325510 31653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:09:34.327158 31653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:09:34.328424 31653 hybrid_clock.cc:648] HybridClock initialized: now 1786518574328367 us; error 49 us; skew 500 ppm
I20260812 07:09:34.328890 31653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:09:34 dist-test-slave-69rg krb5kdc[31637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518574, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:09:34.332463 31653 init.cc:377] Logged in from keytab as kudu/127.6.61.62@KRBTEST.COM (short username kudu)
I20260812 07:09:34.334512 31653 webserver.cc:460] Webserver started at http://127.6.61.62:38603/ using document root <none> and password file <none>
I20260812 07:09:34.335286 31653 fs_manager.cc:359] Metadata directory not provided
I20260812 07:09:34.335450 31653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:09:34.335795 31653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:09:34.339226 31653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "9cc374c54495431d80cb37d223670797"
format_stamp: "Formatted at 2026-08-12 07:09:34 on dist-test-slave-69rg"
server_key: "f59357871ff9432d42b4fe9f2585c45e"
server_key_iv: "0fcdbd39a41d71102b5129471a5e007a"
server_key_version: "encryptionkey@0"
I20260812 07:09:34.340382 31653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "9cc374c54495431d80cb37d223670797"
format_stamp: "Formatted at 2026-08-12 07:09:34 on dist-test-slave-69rg"
server_key: "f59357871ff9432d42b4fe9f2585c45e"
server_key_iv: "0fcdbd39a41d71102b5129471a5e007a"
server_key_version: "encryptionkey@0"
I20260812 07:09:34.346561 31653 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 07:09:34.350580 31671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:09:34.352268 31653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:09:34.352522 31653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "9cc374c54495431d80cb37d223670797"
format_stamp: "Formatted at 2026-08-12 07:09:34 on dist-test-slave-69rg"
server_key: "f59357871ff9432d42b4fe9f2585c45e"
server_key_iv: "0fcdbd39a41d71102b5129471a5e007a"
server_key_version: "encryptionkey@0"
I20260812 07:09:34.352775 31653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:09:34.368618 31653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:09:34.373749 31653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:09:34.374164 31653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:09:34.394830 31653 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:33697
I20260812 07:09:34.394871 31727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:33697 every 8 connection(s)
I20260812 07:09:34.396353 31653 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:09:34.399523 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 31653
I20260812 07:09:34.399755 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:09:34.400225 6388 external_mini_cluster.cc:1428] Setting key dfb97dad35d36907689ed4b50fafee74
I20260812 07:09:34.400574 31728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:09:34.416393 31728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cc374c54495431d80cb37d223670797: Bootstrap starting.
Aug 12 07:09:34 dist-test-slave-69rg krb5kdc[31637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:09:34.419835 31728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cc374c54495431d80cb37d223670797: Neither blocks nor log segments found. Creating new log.
I20260812 07:09:34.420976 31728 log.cc:824] T 00000000000000000000000000000000 P 9cc374c54495431d80cb37d223670797: Log is configured to *not* fsync() on all Append() calls
I20260812 07:09:34.423967 31728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cc374c54495431d80cb37d223670797: No bootstrap required, opened a new log
I20260812 07:09:34.427675 31731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:34.402165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46053 (local address 127.6.61.62:33697)
0812 07:09:34.403037 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:34.403050 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:34.403095 (+ 45us) server_negotiation.cc:407] Connection header received
0812 07:09:34.405431 (+ 2336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:34.405456 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:34.406008 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:34.406369 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:34.408403 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:34.409906 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:34.411867 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:34.412764 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:34.412969 (+ 205us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:34.418709 (+ 5740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:34.418720 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:34.418737 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:34.418783 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:34.422551 (+ 3768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:34.423539 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:34.423547 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:34.423554 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:34.423764 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:34.424400 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:34.424404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:34.424407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:34.425011 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:34.425433 (+ 422us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:34.426146 (+ 713us) server_negotiation.cc:299] Negotiation successful
0812 07:09:34.426505 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":430,"threads_started":1}
I20260812 07:09:34.432144 6388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:09:34.433105 31728 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cc374c54495431d80cb37d223670797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc374c54495431d80cb37d223670797" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 33697 } }
I20260812 07:09:34.433498 31728 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cc374c54495431d80cb37d223670797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:09:34.433605 31728 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cc374c54495431d80cb37d223670797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cc374c54495431d80cb37d223670797, State: Initialized, Role: FOLLOWER
I20260812 07:09:34.434222 31728 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cc374c54495431d80cb37d223670797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc374c54495431d80cb37d223670797" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 33697 } }
I20260812 07:09:34.434482 31728 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cc374c54495431d80cb37d223670797 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:09:34.434625 31728 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cc374c54495431d80cb37d223670797 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:09:34.434805 31728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cc374c54495431d80cb37d223670797 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:09:34.437413 31728 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cc374c54495431d80cb37d223670797 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc374c54495431d80cb37d223670797" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 33697 } }
I20260812 07:09:34.438032 31728 leader_election.cc:302] T 00000000000000000000000000000000 P 9cc374c54495431d80cb37d223670797 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cc374c54495431d80cb37d223670797; no voters:
I20260812 07:09:34.438736 31728 leader_election.cc:288] T 00000000000000000000000000000000 P 9cc374c54495431d80cb37d223670797 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:09:34.438864 31739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cc374c54495431d80cb37d223670797 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:09:34.440068 31739 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cc374c54495431d80cb37d223670797 [term 1 LEADER]: Becoming Leader. State: Replica: 9cc374c54495431d80cb37d223670797, State: Running, Role: LEADER
I20260812 07:09:34.440760 31739 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cc374c54495431d80cb37d223670797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc374c54495431d80cb37d223670797" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 33697 } }
I20260812 07:09:34.441294 31728 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cc374c54495431d80cb37d223670797 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:09:34.444080 31740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cc374c54495431d80cb37d223670797 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cc374c54495431d80cb37d223670797" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc374c54495431d80cb37d223670797" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 33697 } } }
I20260812 07:09:34.444548 31740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cc374c54495431d80cb37d223670797 [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:34.450367 31741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cc374c54495431d80cb37d223670797 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cc374c54495431d80cb37d223670797. Latest consensus state: current_term: 1 leader_uuid: "9cc374c54495431d80cb37d223670797" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc374c54495431d80cb37d223670797" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 33697 } } }
I20260812 07:09:34.450836 31741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cc374c54495431d80cb37d223670797 [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:34.454064 31731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:34.436871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59451 (local address 127.6.61.62:33697)
0812 07:09:34.437060 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:34.437067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:34.437549 (+ 482us) server_negotiation.cc:407] Connection header received
0812 07:09:34.437725 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:34.437735 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:34.437800 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:34.437956 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:34.439806 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:34.440853 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:34.442476 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:34.443310 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:34.443471 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:34.446861 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:34.446866 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:34.446869 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:34.446901 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:34.450736 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:34.451810 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:34.451816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:34.451819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:34.451970 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:34.452449 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:34.452461 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:34.452463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:34.452811 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:34.453116 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:34.453479 (+ 363us) server_negotiation.cc:299] Negotiation successful
0812 07:09:34.453664 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 07:09:34.456446 31746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:09:34.463500 31746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:09:34.474895 31746 catalog_manager.cc:1347] Generated new cluster ID: 51fba6f3afb440a5b44fd45b4d058dde
I20260812 07:09:34.475051 31746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:09:34.487550 31746 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:09:34.489584 31746 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:09:34.499485 31746 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cc374c54495431d80cb37d223670797: Generated new TSK 0
I20260812 07:09:34.500667 31746 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:09:34.508622 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 31653
I20260812 07:09:34.509963 31723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:09:34.603243 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 31653
W20260812 07:09:34.625883 31634 connection.cc:535] client connection to 127.6.61.62:33697 recv error: Network error: failed to read from TLS socket (remote: 127.6.61.62:33697): Connection reset by peer (error 104)
2026-08-12T07:09:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:09:34 dist-test-slave-69rg krb5kdc[31763](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:09:34 dist-test-slave-69rg krb5kdc[31763](info): set up 2 sockets
Aug 12 07:09:34 dist-test-slave-69rg krb5kdc[31763](info): commencing operation
krb5kdc: starting...
W20260812 07:09:35.865433 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.220s user 0.003s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:09:36 dist-test-slave-69rg krb5kdc[31763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:09:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:09:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
I20260812 07:09:36.233609 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:34085
--webserver_interface=127.6.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:33685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:34085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:09:36.601933 31777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:09:36.602283 31777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:09:36.602381 31777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:09:36.609769 31777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:09:36.609973 31777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:09:36.610030 31777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:09:36.610101 31777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:09:36.610140 31777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:09:36.620369 31777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:33685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:34085
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:34085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:09:36.621677 31777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:09:36.623020 31777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:09:36.635862 31788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:36.635792 31784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:36.635831 31786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:09:36.636895 31777 server_base.cc:1034] running on GCE node
I20260812 07:09:36.637768 31777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:09:36.639410 31777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:09:36.640651 31777 hybrid_clock.cc:648] HybridClock initialized: now 1786518576640596 us; error 54 us; skew 500 ppm
I20260812 07:09:36.641036 31777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:09:36 dist-test-slave-69rg krb5kdc[31763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518576, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:09:36.651554 31777 init.cc:377] Logged in from keytab as kudu/127.6.61.62@KRBTEST.COM (short username kudu)
I20260812 07:09:36.653308 31777 webserver.cc:460] Webserver started at http://127.6.61.62:41417/ using document root <none> and password file <none>
I20260812 07:09:36.654080 31777 fs_manager.cc:359] Metadata directory not provided
I20260812 07:09:36.654192 31777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:09:36.654469 31777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:09:36.657585 31777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "2a581fa691314a55a207d21018c35a7a"
format_stamp: "Formatted at 2026-08-12 07:09:36 on dist-test-slave-69rg"
server_key: "7f0ff5ea68509f2263efefcd125d0034"
server_key_iv: "29b65d71d5c3dce83ab803150890a38f"
server_key_version: "encryptionkey@0"
I20260812 07:09:36.658439 31777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "2a581fa691314a55a207d21018c35a7a"
format_stamp: "Formatted at 2026-08-12 07:09:36 on dist-test-slave-69rg"
server_key: "7f0ff5ea68509f2263efefcd125d0034"
server_key_iv: "29b65d71d5c3dce83ab803150890a38f"
server_key_version: "encryptionkey@0"
I20260812 07:09:36.663894 31777 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 07:09:36.667578 31796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:09:36.669116 31777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 07:09:36.669330 31777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "2a581fa691314a55a207d21018c35a7a"
format_stamp: "Formatted at 2026-08-12 07:09:36 on dist-test-slave-69rg"
server_key: "7f0ff5ea68509f2263efefcd125d0034"
server_key_iv: "29b65d71d5c3dce83ab803150890a38f"
server_key_version: "encryptionkey@0"
I20260812 07:09:36.669540 31777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:09:36.697271 31777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:09:36.702171 31777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:09:36.702558 31777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:09:36.722636 31777 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:34085
I20260812 07:09:36.722667 31857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:34085 every 8 connection(s)
I20260812 07:09:36.724061 31777 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:09:36.728078 31858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:09:36.731032 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 31777
I20260812 07:09:36.731221 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:09:36.731771 6388 external_mini_cluster.cc:1428] Setting key 5525dfc0427ab50849c5c5e738772a1e
I20260812 07:09:36.741050 31858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a581fa691314a55a207d21018c35a7a: Bootstrap starting.
I20260812 07:09:36.745235 31858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a581fa691314a55a207d21018c35a7a: Neither blocks nor log segments found. Creating new log.
I20260812 07:09:36.746526 31858 log.cc:824] T 00000000000000000000000000000000 P 2a581fa691314a55a207d21018c35a7a: Log is configured to *not* fsync() on all Append() calls
I20260812 07:09:36.746534 31862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:36.733726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44677 (local address 127.6.61.62:34085)
0812 07:09:36.734686 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:36.734699 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:36.734741 (+ 42us) server_negotiation.cc:407] Connection header received
0812 07:09:36.736392 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:36.736417 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:36.736850 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:36.737213 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:36.737970 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:36.739442 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:36.741400 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:36.742214 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:36.742364 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:36.743927 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:36.743936 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:36.743952 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:09:36.744546 (+ 594us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:36.744631 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:09:36.744659 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:36.744788 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:36.745168 (+ 380us) server_negotiation.cc:299] Negotiation successful
0812 07:09:36.745380 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":467,"threads_started":1}
I20260812 07:09:36.749980 31858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a581fa691314a55a207d21018c35a7a: No bootstrap required, opened a new log
I20260812 07:09:36.751056 6388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:09:36.758906 31858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a581fa691314a55a207d21018c35a7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a581fa691314a55a207d21018c35a7a" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34085 } }
I20260812 07:09:36.759371 31858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a581fa691314a55a207d21018c35a7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:09:36.759487 31858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a581fa691314a55a207d21018c35a7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a581fa691314a55a207d21018c35a7a, State: Initialized, Role: FOLLOWER
I20260812 07:09:36.760222 31858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a581fa691314a55a207d21018c35a7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a581fa691314a55a207d21018c35a7a" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34085 } }
I20260812 07:09:36.760555 31858 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a581fa691314a55a207d21018c35a7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:09:36.760713 31858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a581fa691314a55a207d21018c35a7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:09:36.760919 31858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a581fa691314a55a207d21018c35a7a [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:09:36.763300 31862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:36.755690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53929 (local address 127.6.61.62:34085)
0812 07:09:36.755918 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:36.755925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:36.756434 (+ 509us) server_negotiation.cc:407] Connection header received
0812 07:09:36.756594 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:36.756599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:36.756695 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:36.756855 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:36.757383 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:36.758559 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:36.760174 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:36.760921 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:36.761067 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:36.762398 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:36.762405 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:36.762410 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:09:36.762437 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:36.762505 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:09:36.762528 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:36.762653 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:36.762905 (+ 252us) server_negotiation.cc:299] Negotiation successful
0812 07:09:36.762968 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 07:09:36.764299 31858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a581fa691314a55a207d21018c35a7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a581fa691314a55a207d21018c35a7a" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34085 } }
I20260812 07:09:36.764873 31858 leader_election.cc:302] T 00000000000000000000000000000000 P 2a581fa691314a55a207d21018c35a7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a581fa691314a55a207d21018c35a7a; no voters:
I20260812 07:09:36.765506 31858 leader_election.cc:288] T 00000000000000000000000000000000 P 2a581fa691314a55a207d21018c35a7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:09:36.765663 31871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a581fa691314a55a207d21018c35a7a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:09:36.766525 31871 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a581fa691314a55a207d21018c35a7a [term 1 LEADER]: Becoming Leader. State: Replica: 2a581fa691314a55a207d21018c35a7a, State: Running, Role: LEADER
I20260812 07:09:36.767135 31871 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a581fa691314a55a207d21018c35a7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a581fa691314a55a207d21018c35a7a" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34085 } }
I20260812 07:09:36.767803 31858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a581fa691314a55a207d21018c35a7a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:09:36.770000 31873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a581fa691314a55a207d21018c35a7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a581fa691314a55a207d21018c35a7a. Latest consensus state: current_term: 1 leader_uuid: "2a581fa691314a55a207d21018c35a7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a581fa691314a55a207d21018c35a7a" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34085 } } }
I20260812 07:09:36.770262 31872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a581fa691314a55a207d21018c35a7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a581fa691314a55a207d21018c35a7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a581fa691314a55a207d21018c35a7a" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 34085 } } }
I20260812 07:09:36.770526 31873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a581fa691314a55a207d21018c35a7a [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:36.770654 31872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a581fa691314a55a207d21018c35a7a [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:36.771541 31877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:09:36.785372 31877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:09:36.796640 31877 catalog_manager.cc:1347] Generated new cluster ID: ef0ec2f906024bb8a790ea4bcc5570eb
I20260812 07:09:36.796792 31877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:09:36.811267 31877 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:09:36.812750 31877 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:09:36.819845 31877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a581fa691314a55a207d21018c35a7a: Generated new TSK 0
I20260812 07:09:36.820686 31877 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:09:36.831594 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 31777
W20260812 07:09:36.833139 31853 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44677
W20260812 07:09:36.833961 6388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 31777: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 07:09:36.834132 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 31777
W20260812 07:09:36.856343 31761 connection.cc:535] client connection to 127.6.61.62:34085 recv error: Network error: failed to read from TLS socket (remote: 127.6.61.62:34085): Connection reset by peer (error 104)
2026-08-12T07:09:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2230 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:09:36 dist-test-slave-69rg krb5kdc[31894](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:09:36 dist-test-slave-69rg krb5kdc[31894](info): set up 2 sockets
Aug 12 07:09:36 dist-test-slave-69rg krb5kdc[31894](info): commencing operation
krb5kdc: starting...
W20260812 07:09:38.327919 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.452s user 0.002s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:09:38 dist-test-slave-69rg krb5kdc[31894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:09:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:09:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
W20260812 07:09:38.857710 6388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.6.61.62: real 0.109s user 0.001s sys 0.039s
I20260812 07:09:38.859364 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:35383
--webserver_interface=127.6.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:39921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:35383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:09:39.337453 31909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:09:39.337749 31909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:09:39.337937 31909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:09:39.346038 31909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:09:39.346191 31909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:09:39.346313 31909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:09:39.346403 31909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:09:39.346444 31909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:09:39.356770 31909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:39921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:35383
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:35383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:09:39.358227 31909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:09:39.359599 31909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:09:39.372773 31917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:39.372905 31915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:39.373195 31914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:09:39.373751 31909 server_base.cc:1034] running on GCE node
I20260812 07:09:39.374660 31909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:09:39.376278 31909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:09:39.377552 31909 hybrid_clock.cc:648] HybridClock initialized: now 1786518579377491 us; error 43 us; skew 500 ppm
I20260812 07:09:39.378036 31909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:09:39 dist-test-slave-69rg krb5kdc[31894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518579, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:09:39.381471 31909 init.cc:377] Logged in from keytab as kudu/127.6.61.62@KRBTEST.COM (short username kudu)
I20260812 07:09:39.383246 31909 webserver.cc:460] Webserver started at http://127.6.61.62:37303/ using document root <none> and password file <none>
I20260812 07:09:39.383985 31909 fs_manager.cc:359] Metadata directory not provided
I20260812 07:09:39.384143 31909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:09:39.384479 31909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:09:39.387809 31909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "c2c3a01f913e43d6a81d1de083edfe14"
format_stamp: "Formatted at 2026-08-12 07:09:39 on dist-test-slave-69rg"
server_key: "3e9eb7c90bc5cb0210410af05e34c7c0"
server_key_iv: "31a7a928ec1df15be6657946d52d56d9"
server_key_version: "encryptionkey@0"
I20260812 07:09:39.388656 31909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "c2c3a01f913e43d6a81d1de083edfe14"
format_stamp: "Formatted at 2026-08-12 07:09:39 on dist-test-slave-69rg"
server_key: "3e9eb7c90bc5cb0210410af05e34c7c0"
server_key_iv: "31a7a928ec1df15be6657946d52d56d9"
server_key_version: "encryptionkey@0"
I20260812 07:09:39.394356 31909 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260812 07:09:39.398350 31925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:09:39.399930 31909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 07:09:39.400143 31909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "c2c3a01f913e43d6a81d1de083edfe14"
format_stamp: "Formatted at 2026-08-12 07:09:39 on dist-test-slave-69rg"
server_key: "3e9eb7c90bc5cb0210410af05e34c7c0"
server_key_iv: "31a7a928ec1df15be6657946d52d56d9"
server_key_version: "encryptionkey@0"
I20260812 07:09:39.400386 31909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:09:39.406746 31909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:09:39.411298 31909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:09:39.411623 31909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:09:39.431609 31909 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:35383
I20260812 07:09:39.431706 31981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:35383 every 8 connection(s)
I20260812 07:09:39.433080 31909 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:09:39.433890 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 31909
I20260812 07:09:39.434085 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:09:39.434645 6388 external_mini_cluster.cc:1428] Setting key 14b49de321efe1283a6b20da741eedea
I20260812 07:09:39.438118 31982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:09:39.444216 31985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:39.436704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46327 (local address 127.6.61.62:35383)
0812 07:09:39.437562 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:39.437581 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:39.437636 (+ 55us) server_negotiation.cc:407] Connection header received
0812 07:09:39.440020 (+ 2384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:39.440053 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:39.440500 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:39.440948 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:39.442165 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:39.442174 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:39.442192 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:09:39.442218 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:39.442310 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:09:39.442339 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:39.442567 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:39.442722 (+ 155us) server_negotiation.cc:299] Negotiation successful
0812 07:09:39.442932 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":336,"threads_started":1}
I20260812 07:09:39.448212 6388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:09:39.451153 31982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2c3a01f913e43d6a81d1de083edfe14: Bootstrap starting.
I20260812 07:09:39.455034 31982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2c3a01f913e43d6a81d1de083edfe14: Neither blocks nor log segments found. Creating new log.
I20260812 07:09:39.455741 31985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:39.452698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45185 (local address 127.6.61.62:35383)
0812 07:09:39.452963 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:39.452972 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:39.453395 (+ 423us) server_negotiation.cc:407] Connection header received
0812 07:09:39.453630 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:39.453634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:39.453686 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:39.453786 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:39.454919 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:39.454923 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:39.454927 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:09:39.454948 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:39.454987 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:09:39.455002 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:39.455108 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:39.455345 (+ 237us) server_negotiation.cc:299] Negotiation successful
0812 07:09:39.455402 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 07:09:39.456405 31982 log.cc:824] T 00000000000000000000000000000000 P c2c3a01f913e43d6a81d1de083edfe14: Log is configured to *not* fsync() on all Append() calls
I20260812 07:09:39.459316 31982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2c3a01f913e43d6a81d1de083edfe14: No bootstrap required, opened a new log
I20260812 07:09:39.466717 31982 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2c3a01f913e43d6a81d1de083edfe14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2c3a01f913e43d6a81d1de083edfe14" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35383 } }
I20260812 07:09:39.467149 31982 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2c3a01f913e43d6a81d1de083edfe14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:09:39.467298 31982 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2c3a01f913e43d6a81d1de083edfe14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2c3a01f913e43d6a81d1de083edfe14, State: Initialized, Role: FOLLOWER
I20260812 07:09:39.467886 31982 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2c3a01f913e43d6a81d1de083edfe14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2c3a01f913e43d6a81d1de083edfe14" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35383 } }
I20260812 07:09:39.468140 31982 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2c3a01f913e43d6a81d1de083edfe14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:09:39.468295 31982 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2c3a01f913e43d6a81d1de083edfe14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:09:39.468472 31982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2c3a01f913e43d6a81d1de083edfe14 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:09:39.471148 31982 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2c3a01f913e43d6a81d1de083edfe14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2c3a01f913e43d6a81d1de083edfe14" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35383 } }
I20260812 07:09:39.471697 31982 leader_election.cc:302] T 00000000000000000000000000000000 P c2c3a01f913e43d6a81d1de083edfe14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2c3a01f913e43d6a81d1de083edfe14; no voters:
I20260812 07:09:39.472358 31982 leader_election.cc:288] T 00000000000000000000000000000000 P c2c3a01f913e43d6a81d1de083edfe14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:09:39.472543 31994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2c3a01f913e43d6a81d1de083edfe14 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:09:39.473701 31994 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2c3a01f913e43d6a81d1de083edfe14 [term 1 LEADER]: Becoming Leader. State: Replica: c2c3a01f913e43d6a81d1de083edfe14, State: Running, Role: LEADER
I20260812 07:09:39.474342 31994 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2c3a01f913e43d6a81d1de083edfe14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2c3a01f913e43d6a81d1de083edfe14" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35383 } }
I20260812 07:09:39.474979 31982 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2c3a01f913e43d6a81d1de083edfe14 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:09:39.477245 31995 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2c3a01f913e43d6a81d1de083edfe14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2c3a01f913e43d6a81d1de083edfe14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2c3a01f913e43d6a81d1de083edfe14" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35383 } } }
I20260812 07:09:39.477679 31995 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2c3a01f913e43d6a81d1de083edfe14 [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:39.479071 31996 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2c3a01f913e43d6a81d1de083edfe14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2c3a01f913e43d6a81d1de083edfe14. Latest consensus state: current_term: 1 leader_uuid: "c2c3a01f913e43d6a81d1de083edfe14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2c3a01f913e43d6a81d1de083edfe14" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35383 } } }
I20260812 07:09:39.479480 31996 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2c3a01f913e43d6a81d1de083edfe14 [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:39.480808 32000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:09:39.486913 32000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:09:39.496016 32000 catalog_manager.cc:1347] Generated new cluster ID: bfd8962ed0b84400984ee42ffba13c1f
I20260812 07:09:39.496153 32000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:09:39.510636 32000 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:09:39.511101 32000 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:09:39.519156 32000 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2c3a01f913e43d6a81d1de083edfe14: Generated new TSK 0
I20260812 07:09:39.520068 32000 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:09:39.533397 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 31909
W20260812 07:09:39.534628 31975 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46327
W20260812 07:09:39.535363 6388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 31909: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 07:09:39.535522 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 31909
2026-08-12T07:09:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2703 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:09:39 dist-test-slave-69rg krb5kdc[32017](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:09:39 dist-test-slave-69rg krb5kdc[32017](info): set up 2 sockets
Aug 12 07:09:39 dist-test-slave-69rg krb5kdc[32017](info): commencing operation
krb5kdc: starting...
W20260812 07:09:40.805903 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.227s user 0.006s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:09:41 dist-test-slave-69rg krb5kdc[32017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:09:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:09:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
I20260812 07:09:41.170953 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:32807
--webserver_interface=127.6.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:35755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:32807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:09:41.543444 32032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:09:41.543756 32032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:09:41.543921 32032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:09:41.552029 32032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:09:41.552174 32032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:09:41.552290 32032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:09:41.552376 32032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:09:41.552459 32032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:09:41.563277 32032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:35755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:32807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:32807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:09:41.564680 32032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:09:41.566120 32032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:09:41.578685 32039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:41.578768 32040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:41.578806 32042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:09:41.579900 32032 server_base.cc:1034] running on GCE node
I20260812 07:09:41.580780 32032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:09:41.582340 32032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:09:41.583609 32032 hybrid_clock.cc:648] HybridClock initialized: now 1786518581583555 us; error 38 us; skew 500 ppm
I20260812 07:09:41.584036 32032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:09:41 dist-test-slave-69rg krb5kdc[32017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518581, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:09:41.587560 32032 init.cc:377] Logged in from keytab as kudu/127.6.61.62@KRBTEST.COM (short username kudu)
I20260812 07:09:41.589284 32032 webserver.cc:460] Webserver started at http://127.6.61.62:41233/ using document root <none> and password file <none>
I20260812 07:09:41.590126 32032 fs_manager.cc:359] Metadata directory not provided
I20260812 07:09:41.590282 32032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:09:41.590621 32032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:09:41.593941 32032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "b3181631088545899b2d0012dcfa16d5"
format_stamp: "Formatted at 2026-08-12 07:09:41 on dist-test-slave-69rg"
server_key: "9938fbb8a96ec096f9778684e95ced1b"
server_key_iv: "8b6df20b2bdf08b8934bca264d29b14f"
server_key_version: "encryptionkey@0"
I20260812 07:09:41.594858 32032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "b3181631088545899b2d0012dcfa16d5"
format_stamp: "Formatted at 2026-08-12 07:09:41 on dist-test-slave-69rg"
server_key: "9938fbb8a96ec096f9778684e95ced1b"
server_key_iv: "8b6df20b2bdf08b8934bca264d29b14f"
server_key_version: "encryptionkey@0"
I20260812 07:09:41.600749 32032 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260812 07:09:41.604741 32050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:09:41.606297 32032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:09:41.606523 32032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "b3181631088545899b2d0012dcfa16d5"
format_stamp: "Formatted at 2026-08-12 07:09:41 on dist-test-slave-69rg"
server_key: "9938fbb8a96ec096f9778684e95ced1b"
server_key_iv: "8b6df20b2bdf08b8934bca264d29b14f"
server_key_version: "encryptionkey@0"
I20260812 07:09:41.606765 32032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:09:41.626530 32032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:09:41.632339 32032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:09:41.632684 32032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:09:41.653631 32032 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:32807
I20260812 07:09:41.653692 32107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:32807 every 8 connection(s)
I20260812 07:09:41.655241 32032 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:09:41.658631 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 32032
I20260812 07:09:41.658833 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:09:41.659327 6388 external_mini_cluster.cc:1428] Setting key b312d1928344eabcd35dacaec376c731
I20260812 07:09:41.659551 32110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:09:41.674726 32110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3181631088545899b2d0012dcfa16d5: Bootstrap starting.
I20260812 07:09:41.678658 32110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3181631088545899b2d0012dcfa16d5: Neither blocks nor log segments found. Creating new log.
I20260812 07:09:41.680186 32110 log.cc:824] T 00000000000000000000000000000000 P b3181631088545899b2d0012dcfa16d5: Log is configured to *not* fsync() on all Append() calls
Aug 12 07:09:41 dist-test-slave-69rg krb5kdc[32017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:09:41.684026 32110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3181631088545899b2d0012dcfa16d5: No bootstrap required, opened a new log
I20260812 07:09:41.690544 32113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:41.661307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36875 (local address 127.6.61.62:32807)
0812 07:09:41.662179 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:41.662190 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:41.662235 (+ 45us) server_negotiation.cc:407] Connection header received
0812 07:09:41.664466 (+ 2231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:41.664493 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:41.664959 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:41.665320 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:41.667421 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:41.668768 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:41.670781 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:41.671628 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:41.671802 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:41.682240 (+ 10438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:41.682249 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:41.682267 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:41.682307 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:41.685700 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:41.686642 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:41.686650 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:41.686661 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:41.686868 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:41.687311 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:41.687316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:41.687318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:41.687947 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:41.688290 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:41.688815 (+ 525us) server_negotiation.cc:299] Negotiation successful
0812 07:09:41.689310 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":428,"threads_started":1}
I20260812 07:09:41.693254 32110 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3181631088545899b2d0012dcfa16d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3181631088545899b2d0012dcfa16d5" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 32807 } }
I20260812 07:09:41.693748 32110 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3181631088545899b2d0012dcfa16d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:09:41.693924 32110 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3181631088545899b2d0012dcfa16d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3181631088545899b2d0012dcfa16d5, State: Initialized, Role: FOLLOWER
I20260812 07:09:41.694898 6388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:09:41.694773 32110 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3181631088545899b2d0012dcfa16d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3181631088545899b2d0012dcfa16d5" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 32807 } }
I20260812 07:09:41.695073 32110 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3181631088545899b2d0012dcfa16d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:09:41.695255 32110 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3181631088545899b2d0012dcfa16d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:09:41.695457 32110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3181631088545899b2d0012dcfa16d5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:09:41.698350 32110 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3181631088545899b2d0012dcfa16d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3181631088545899b2d0012dcfa16d5" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 32807 } }
I20260812 07:09:41.698943 32110 leader_election.cc:302] T 00000000000000000000000000000000 P b3181631088545899b2d0012dcfa16d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3181631088545899b2d0012dcfa16d5; no voters:
I20260812 07:09:41.699764 32110 leader_election.cc:288] T 00000000000000000000000000000000 P b3181631088545899b2d0012dcfa16d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:09:41.699918 32120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3181631088545899b2d0012dcfa16d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:09:41.703035 32110 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3181631088545899b2d0012dcfa16d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:09:41.707042 32120 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3181631088545899b2d0012dcfa16d5 [term 1 LEADER]: Becoming Leader. State: Replica: b3181631088545899b2d0012dcfa16d5, State: Running, Role: LEADER
I20260812 07:09:41.708079 32120 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3181631088545899b2d0012dcfa16d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3181631088545899b2d0012dcfa16d5" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 32807 } }
I20260812 07:09:41.712287 32122 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3181631088545899b2d0012dcfa16d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3181631088545899b2d0012dcfa16d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3181631088545899b2d0012dcfa16d5" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 32807 } } }
I20260812 07:09:41.712927 32122 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3181631088545899b2d0012dcfa16d5 [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:41.716960 32123 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3181631088545899b2d0012dcfa16d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3181631088545899b2d0012dcfa16d5. Latest consensus state: current_term: 1 leader_uuid: "b3181631088545899b2d0012dcfa16d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3181631088545899b2d0012dcfa16d5" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 32807 } } }
I20260812 07:09:41.717348 32123 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3181631088545899b2d0012dcfa16d5 [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:41.718181 32130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:09:41.724491 32113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:41.699762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54831 (local address 127.6.61.62:32807)
0812 07:09:41.700034 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:41.700044 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:41.706759 (+ 6715us) server_negotiation.cc:407] Connection header received
0812 07:09:41.706989 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:41.706993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:41.707071 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:41.707227 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:41.708916 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:41.709680 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:41.711566 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:41.712292 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:41.712414 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:41.716555 (+ 4141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:41.716564 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:41.716570 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:41.716616 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:41.720835 (+ 4219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:41.721846 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:41.721852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:41.721856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:41.722046 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:41.722534 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:41.722540 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:41.722544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:41.723142 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:41.723483 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:41.723840 (+ 357us) server_negotiation.cc:299] Negotiation successful
0812 07:09:41.724111 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260812 07:09:41.725665 32130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:09:41.736689 32130 catalog_manager.cc:1347] Generated new cluster ID: 08bf4fc67d0943ffb4250385c1ec0e02
I20260812 07:09:41.736856 32130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:09:41.765808 32130 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:09:41.767282 32130 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:09:41.773303 32130 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3181631088545899b2d0012dcfa16d5: Generated new TSK 0
I20260812 07:09:41.774407 32130 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:09:41.786768 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 32032
I20260812 07:09:41.788028 32103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:09:41.888051 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 32032
2026-08-12T07:09:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2352 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:09:42 dist-test-slave-69rg krb5kdc[32148](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:09:42 dist-test-slave-69rg krb5kdc[32148](info): set up 2 sockets
Aug 12 07:09:42 dist-test-slave-69rg krb5kdc[32148](info): commencing operation
krb5kdc: starting...
W20260812 07:09:43.153262 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.221s user 0.006s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:09:43 dist-test-slave-69rg krb5kdc[32148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:09:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:09:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
I20260812 07:09:43.515924 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:37239
--webserver_interface=127.6.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:45745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:37239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:09:43.873018 32166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:09:43.873431 32166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:09:43.873615 32166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:09:43.881302 32166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:09:43.881484 32166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:09:43.881593 32166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:09:43.881704 32166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:09:43.881796 32166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:09:43.892135 32166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:45745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:37239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:37239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:09:43.893573 32166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:09:43.894997 32166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:09:43.907707 32172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:43.907809 32174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:43.907833 32171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:09:43.908879 32166 server_base.cc:1034] running on GCE node
I20260812 07:09:43.909725 32166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:09:43.911345 32166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:09:43.912578 32166 hybrid_clock.cc:648] HybridClock initialized: now 1786518583912529 us; error 47 us; skew 500 ppm
I20260812 07:09:43.912966 32166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:09:43 dist-test-slave-69rg krb5kdc[32148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518583, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:09:43.916345 32166 init.cc:377] Logged in from keytab as kudu/127.6.61.62@KRBTEST.COM (short username kudu)
I20260812 07:09:43.918097 32166 webserver.cc:460] Webserver started at http://127.6.61.62:42455/ using document root <none> and password file <none>
I20260812 07:09:43.918850 32166 fs_manager.cc:359] Metadata directory not provided
I20260812 07:09:43.919010 32166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:09:43.919363 32166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:09:43.922715 32166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "8baa77a2b9ed4ed884183e87d8b223c3"
format_stamp: "Formatted at 2026-08-12 07:09:43 on dist-test-slave-69rg"
server_key: "b841aa77fc1c8eb0ea390151f0828d7d"
server_key_iv: "bb2ed9f40ccde12e4893c86c5748de90"
server_key_version: "encryptionkey@0"
I20260812 07:09:43.923537 32166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "8baa77a2b9ed4ed884183e87d8b223c3"
format_stamp: "Formatted at 2026-08-12 07:09:43 on dist-test-slave-69rg"
server_key: "b841aa77fc1c8eb0ea390151f0828d7d"
server_key_iv: "bb2ed9f40ccde12e4893c86c5748de90"
server_key_version: "encryptionkey@0"
I20260812 07:09:43.928990 32166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260812 07:09:43.932901 32182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:09:43.934275 32166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:09:43.934508 32166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "8baa77a2b9ed4ed884183e87d8b223c3"
format_stamp: "Formatted at 2026-08-12 07:09:43 on dist-test-slave-69rg"
server_key: "b841aa77fc1c8eb0ea390151f0828d7d"
server_key_iv: "bb2ed9f40ccde12e4893c86c5748de90"
server_key_version: "encryptionkey@0"
I20260812 07:09:43.934749 32166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:09:43.957465 32166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:09:43.962512 32166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:09:43.962855 32166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:09:43.983500 32166 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:37239
I20260812 07:09:43.983533 32242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:37239 every 8 connection(s)
I20260812 07:09:43.984998 32166 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:09:43.989141 32243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:09:43.991888 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 32166
I20260812 07:09:43.992086 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:09:43.992640 6388 external_mini_cluster.cc:1428] Setting key 926b805dd636a49ac0132b7bdaa8a757
I20260812 07:09:44.002660 32243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8baa77a2b9ed4ed884183e87d8b223c3: Bootstrap starting.
I20260812 07:09:44.006539 32243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8baa77a2b9ed4ed884183e87d8b223c3: Neither blocks nor log segments found. Creating new log.
I20260812 07:09:44.007820 32247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:43.994796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59973 (local address 127.6.61.62:37239)
0812 07:09:43.995724 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:43.995738 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:43.995792 (+ 54us) server_negotiation.cc:407] Connection header received
0812 07:09:43.997511 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:43.997538 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:43.998109 (+ 571us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:43.998561 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:43.999193 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:44.000655 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:44.002206 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:44.003030 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:44.003257 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:44.004660 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:44.004668 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:44.004685 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:09:44.005254 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:44.005417 (+ 163us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:09:44.005461 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:44.005597 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:44.005921 (+ 324us) server_negotiation.cc:299] Negotiation successful
0812 07:09:44.006235 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":415,"threads_started":1}
I20260812 07:09:44.008169 32243 log.cc:824] T 00000000000000000000000000000000 P 8baa77a2b9ed4ed884183e87d8b223c3: Log is configured to *not* fsync() on all Append() calls
I20260812 07:09:44.011507 32243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8baa77a2b9ed4ed884183e87d8b223c3: No bootstrap required, opened a new log
I20260812 07:09:44.012055 6388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:09:44.020462 32243 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8baa77a2b9ed4ed884183e87d8b223c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8baa77a2b9ed4ed884183e87d8b223c3" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 37239 } }
I20260812 07:09:44.021013 32243 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8baa77a2b9ed4ed884183e87d8b223c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:09:44.021183 32243 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8baa77a2b9ed4ed884183e87d8b223c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8baa77a2b9ed4ed884183e87d8b223c3, State: Initialized, Role: FOLLOWER
I20260812 07:09:44.021797 32243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8baa77a2b9ed4ed884183e87d8b223c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8baa77a2b9ed4ed884183e87d8b223c3" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 37239 } }
I20260812 07:09:44.022149 32243 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8baa77a2b9ed4ed884183e87d8b223c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:09:44.022315 32243 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8baa77a2b9ed4ed884183e87d8b223c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:09:44.022511 32243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8baa77a2b9ed4ed884183e87d8b223c3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:09:44.024048 32247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:44.016509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60209 (local address 127.6.61.62:37239)
0812 07:09:44.016747 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:44.016756 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:44.017314 (+ 558us) server_negotiation.cc:407] Connection header received
0812 07:09:44.017496 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:44.017501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:44.017580 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:44.017681 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:44.018388 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:44.019189 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:44.021074 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:44.021778 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:44.021960 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:44.023268 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:44.023273 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:44.023276 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:09:44.023298 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:44.023340 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:09:44.023360 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:44.023438 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:44.023598 (+ 160us) server_negotiation.cc:299] Negotiation successful
0812 07:09:44.023688 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 07:09:44.025609 32243 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8baa77a2b9ed4ed884183e87d8b223c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8baa77a2b9ed4ed884183e87d8b223c3" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 37239 } }
I20260812 07:09:44.026167 32243 leader_election.cc:302] T 00000000000000000000000000000000 P 8baa77a2b9ed4ed884183e87d8b223c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8baa77a2b9ed4ed884183e87d8b223c3; no voters:
I20260812 07:09:44.026831 32243 leader_election.cc:288] T 00000000000000000000000000000000 P 8baa77a2b9ed4ed884183e87d8b223c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:09:44.026975 32255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8baa77a2b9ed4ed884183e87d8b223c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:09:44.027766 32255 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8baa77a2b9ed4ed884183e87d8b223c3 [term 1 LEADER]: Becoming Leader. State: Replica: 8baa77a2b9ed4ed884183e87d8b223c3, State: Running, Role: LEADER
I20260812 07:09:44.028292 32255 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8baa77a2b9ed4ed884183e87d8b223c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8baa77a2b9ed4ed884183e87d8b223c3" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 37239 } }
I20260812 07:09:44.029091 32243 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8baa77a2b9ed4ed884183e87d8b223c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:09:44.031384 32257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8baa77a2b9ed4ed884183e87d8b223c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8baa77a2b9ed4ed884183e87d8b223c3. Latest consensus state: current_term: 1 leader_uuid: "8baa77a2b9ed4ed884183e87d8b223c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8baa77a2b9ed4ed884183e87d8b223c3" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 37239 } } }
I20260812 07:09:44.031816 32257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8baa77a2b9ed4ed884183e87d8b223c3 [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:44.032711 32266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:09:44.033716 32256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8baa77a2b9ed4ed884183e87d8b223c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8baa77a2b9ed4ed884183e87d8b223c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8baa77a2b9ed4ed884183e87d8b223c3" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 37239 } } }
I20260812 07:09:44.034104 32256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8baa77a2b9ed4ed884183e87d8b223c3 [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:44.039142 32266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:09:44.048388 32266 catalog_manager.cc:1347] Generated new cluster ID: c5487111176942e1b22ed985fc3e1c0c
I20260812 07:09:44.048524 32266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:09:44.082017 32266 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:09:44.083405 32266 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:09:44.090232 32266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8baa77a2b9ed4ed884183e87d8b223c3: Generated new TSK 0
I20260812 07:09:44.091288 32266 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:09:44.107542 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 32166
W20260812 07:09:44.108871 32238 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59973
W20260812 07:09:44.109721 6388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 32166: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 07:09:44.109928 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 32166
2026-08-12T07:09:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2219 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:09:44 dist-test-slave-69rg krb5kdc[32279](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:09:44 dist-test-slave-69rg krb5kdc[32279](info): set up 2 sockets
Aug 12 07:09:44 dist-test-slave-69rg krb5kdc[32279](info): commencing operation
krb5kdc: starting...
W20260812 07:09:45.349431 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.198s user 0.000s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:09:45 dist-test-slave-69rg krb5kdc[32279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:09:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:09:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
I20260812 07:09:45.718392 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:42869
--webserver_interface=127.6.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:44649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:42869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:09:46.091382 32293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:09:46.091684 32293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:09:46.091840 32293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:09:46.099409 32293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:09:46.099541 32293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:09:46.099660 32293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:09:46.099808 32293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:09:46.099886 32293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:09:46.110391 32293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:44649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:42869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:42869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:09:46.111876 32293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:09:46.113235 32293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:09:46.125399 32299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:46.125381 32300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:46.125474 32302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:09:46.126821 32293 server_base.cc:1034] running on GCE node
I20260812 07:09:46.127719 32293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:09:46.129120 32293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:09:46.130326 32293 hybrid_clock.cc:648] HybridClock initialized: now 1786518586130275 us; error 37 us; skew 500 ppm
I20260812 07:09:46.130698 32293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:09:46 dist-test-slave-69rg krb5kdc[32279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518586, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:09:46.134119 32293 init.cc:377] Logged in from keytab as kudu/127.6.61.62@KRBTEST.COM (short username kudu)
I20260812 07:09:46.135728 32293 webserver.cc:460] Webserver started at http://127.6.61.62:39545/ using document root <none> and password file <none>
I20260812 07:09:46.136439 32293 fs_manager.cc:359] Metadata directory not provided
I20260812 07:09:46.136595 32293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:09:46.136927 32293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:09:46.140089 32293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "c0bf56d4bc7743c2b00f731e76af5eb9"
format_stamp: "Formatted at 2026-08-12 07:09:46 on dist-test-slave-69rg"
server_key: "bc9cdc2ddb51449c3325a1fb6e9e3bed"
server_key_iv: "0b838751cb028bd98216f3a7018216b7"
server_key_version: "encryptionkey@0"
I20260812 07:09:46.140919 32293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "c0bf56d4bc7743c2b00f731e76af5eb9"
format_stamp: "Formatted at 2026-08-12 07:09:46 on dist-test-slave-69rg"
server_key: "bc9cdc2ddb51449c3325a1fb6e9e3bed"
server_key_iv: "0b838751cb028bd98216f3a7018216b7"
server_key_version: "encryptionkey@0"
I20260812 07:09:46.146688 32293 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 07:09:46.150619 32311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:09:46.152252 32293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:09:46.152488 32293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "c0bf56d4bc7743c2b00f731e76af5eb9"
format_stamp: "Formatted at 2026-08-12 07:09:46 on dist-test-slave-69rg"
server_key: "bc9cdc2ddb51449c3325a1fb6e9e3bed"
server_key_iv: "0b838751cb028bd98216f3a7018216b7"
server_key_version: "encryptionkey@0"
I20260812 07:09:46.152743 32293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:09:46.159330 32293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:09:46.164144 32293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:09:46.164496 32293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:09:46.184895 32293 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:42869
I20260812 07:09:46.184947 32370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:42869 every 8 connection(s)
I20260812 07:09:46.186532 32293 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:09:46.190415 32371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:09:46.193668 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 32293
I20260812 07:09:46.193928 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:09:46.194403 6388 external_mini_cluster.cc:1428] Setting key 96b6f607f17b6eb6190f8bd144b411c7
I20260812 07:09:46.202813 32374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:46.196442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45827 (local address 127.6.61.62:42869)
0812 07:09:46.197275 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:46.197286 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:46.197328 (+ 42us) server_negotiation.cc:407] Connection header received
0812 07:09:46.199115 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:46.199141 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:46.199527 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:46.199866 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:46.201041 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:46.201048 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:46.201061 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:09:46.201076 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:46.201141 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:09:46.201161 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:46.201386 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:46.201553 (+ 167us) server_negotiation.cc:299] Negotiation successful
0812 07:09:46.201774 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":381,"threads_started":1}
I20260812 07:09:46.203509 32371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0bf56d4bc7743c2b00f731e76af5eb9: Bootstrap starting.
I20260812 07:09:46.206643 6388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:09:46.206947 32371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0bf56d4bc7743c2b00f731e76af5eb9: Neither blocks nor log segments found. Creating new log.
I20260812 07:09:46.208249 32371 log.cc:824] T 00000000000000000000000000000000 P c0bf56d4bc7743c2b00f731e76af5eb9: Log is configured to *not* fsync() on all Append() calls
I20260812 07:09:46.211920 32371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0bf56d4bc7743c2b00f731e76af5eb9: No bootstrap required, opened a new log
I20260812 07:09:46.215162 32374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:46.212215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49541 (local address 127.6.61.62:42869)
0812 07:09:46.212486 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:46.212496 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:46.212915 (+ 419us) server_negotiation.cc:407] Connection header received
0812 07:09:46.213102 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:46.213105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:46.213154 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:46.213245 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:46.214431 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:46.214436 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:46.214447 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:09:46.214493 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:46.214531 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:09:46.214546 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:46.214643 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:46.214771 (+ 128us) server_negotiation.cc:299] Negotiation successful
0812 07:09:46.214818 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260812 07:09:46.220770 32371 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0bf56d4bc7743c2b00f731e76af5eb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0bf56d4bc7743c2b00f731e76af5eb9" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 42869 } }
I20260812 07:09:46.221194 32371 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0bf56d4bc7743c2b00f731e76af5eb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:09:46.221311 32371 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0bf56d4bc7743c2b00f731e76af5eb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0bf56d4bc7743c2b00f731e76af5eb9, State: Initialized, Role: FOLLOWER
I20260812 07:09:46.222025 32371 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0bf56d4bc7743c2b00f731e76af5eb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0bf56d4bc7743c2b00f731e76af5eb9" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 42869 } }
I20260812 07:09:46.222276 32371 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0bf56d4bc7743c2b00f731e76af5eb9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:09:46.222434 32371 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0bf56d4bc7743c2b00f731e76af5eb9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:09:46.222621 32371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0bf56d4bc7743c2b00f731e76af5eb9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:09:46.224824 32371 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0bf56d4bc7743c2b00f731e76af5eb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0bf56d4bc7743c2b00f731e76af5eb9" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 42869 } }
I20260812 07:09:46.225347 32371 leader_election.cc:302] T 00000000000000000000000000000000 P c0bf56d4bc7743c2b00f731e76af5eb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0bf56d4bc7743c2b00f731e76af5eb9; no voters:
I20260812 07:09:46.226133 32371 leader_election.cc:288] T 00000000000000000000000000000000 P c0bf56d4bc7743c2b00f731e76af5eb9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:09:46.226209 32381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0bf56d4bc7743c2b00f731e76af5eb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:09:46.226986 32381 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0bf56d4bc7743c2b00f731e76af5eb9 [term 1 LEADER]: Becoming Leader. State: Replica: c0bf56d4bc7743c2b00f731e76af5eb9, State: Running, Role: LEADER
I20260812 07:09:46.227600 32381 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0bf56d4bc7743c2b00f731e76af5eb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0bf56d4bc7743c2b00f731e76af5eb9" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 42869 } }
I20260812 07:09:46.228349 32371 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0bf56d4bc7743c2b00f731e76af5eb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:09:46.230715 32382 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0bf56d4bc7743c2b00f731e76af5eb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0bf56d4bc7743c2b00f731e76af5eb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0bf56d4bc7743c2b00f731e76af5eb9" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 42869 } } }
I20260812 07:09:46.231093 32382 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0bf56d4bc7743c2b00f731e76af5eb9 [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:46.231585 32383 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0bf56d4bc7743c2b00f731e76af5eb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0bf56d4bc7743c2b00f731e76af5eb9. Latest consensus state: current_term: 1 leader_uuid: "c0bf56d4bc7743c2b00f731e76af5eb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0bf56d4bc7743c2b00f731e76af5eb9" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 42869 } } }
I20260812 07:09:46.231954 32383 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0bf56d4bc7743c2b00f731e76af5eb9 [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:46.244042 32387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:09:46.251333 32387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:09:46.261938 32387 catalog_manager.cc:1347] Generated new cluster ID: a1d7c8aa05994e548598320bb807c467
I20260812 07:09:46.262089 32387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:09:46.286953 32387 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:09:46.287469 32387 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:09:46.301148 32387 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0bf56d4bc7743c2b00f731e76af5eb9: Generated new TSK 0
I20260812 07:09:46.302161 32387 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:09:46.312055 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 32293
W20260812 07:09:46.313213 32365 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45827
W20260812 07:09:46.313962 6388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 32293: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 07:09:46.314147 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 32293
2026-08-12T07:09:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2202 ms)
[----------] 18 tests from AuthTokenIssuingTest (42997 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:09:46 dist-test-slave-69rg krb5kdc[32404](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:09:46 dist-test-slave-69rg krb5kdc[32404](info): set up 2 sockets
Aug 12 07:09:46 dist-test-slave-69rg krb5kdc[32404](info): commencing operation
krb5kdc: starting...
W20260812 07:09:47.568099 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.214s user 0.007s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:09:47 dist-test-slave-69rg krb5kdc[32404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:09:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:09:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
I20260812 07:09:47.928035 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:36257
--webserver_interface=127.6.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:46551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:36257,127.6.61.61:34701,127.6.61.60:42729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:09:48.308768 32421 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 07:09:48.309129 32421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:09:48.309212 32421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:09:48.309304 32421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:09:48.317330 32421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:09:48.317441 32421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:09:48.317502 32421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:09:48.317545 32421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:09:48.317610 32421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:09:48.327796 32421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:46551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:36257,127.6.61.61:34701,127.6.61.60:42729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:36257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:09:48.329078 32421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:09:48.330411 32421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:09:48.342950 32426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:48.342993 32428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:48.343041 32430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:09:48.347570 32421 server_base.cc:1034] running on GCE node
I20260812 07:09:48.348521 32421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:09:48.350164 32421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:09:48.351441 32421 hybrid_clock.cc:648] HybridClock initialized: now 1786518588351392 us; error 36 us; skew 500 ppm
I20260812 07:09:48.351819 32421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:09:48 dist-test-slave-69rg krb5kdc[32404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518588, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:09:48.359421 32421 init.cc:377] Logged in from keytab as kudu/127.6.61.62@KRBTEST.COM (short username kudu)
I20260812 07:09:48.361133 32421 webserver.cc:460] Webserver started at http://127.6.61.62:38859/ using document root <none> and password file <none>
I20260812 07:09:48.361901 32421 fs_manager.cc:359] Metadata directory not provided
I20260812 07:09:48.362002 32421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:09:48.362293 32421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:09:48.365389 32421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "3bf80683ae714a0cacf072bd357cdaea"
format_stamp: "Formatted at 2026-08-12 07:09:48 on dist-test-slave-69rg"
server_key: "af5919764ec50c119093e3c59543f487"
server_key_iv: "ea6e3ea074007d03a6e02143e63073a7"
server_key_version: "encryptionkey@0"
I20260812 07:09:48.366253 32421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "3bf80683ae714a0cacf072bd357cdaea"
format_stamp: "Formatted at 2026-08-12 07:09:48 on dist-test-slave-69rg"
server_key: "af5919764ec50c119093e3c59543f487"
server_key_iv: "ea6e3ea074007d03a6e02143e63073a7"
server_key_version: "encryptionkey@0"
I20260812 07:09:48.371521 32421 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 07:09:48.375243 32437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:09:48.376703 32421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:09:48.376896 32421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "3bf80683ae714a0cacf072bd357cdaea"
format_stamp: "Formatted at 2026-08-12 07:09:48 on dist-test-slave-69rg"
server_key: "af5919764ec50c119093e3c59543f487"
server_key_iv: "ea6e3ea074007d03a6e02143e63073a7"
server_key_version: "encryptionkey@0"
I20260812 07:09:48.377121 32421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:09:48.388440 32421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:09:48.393671 32421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:09:48.393994 32421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:09:48.413673 32421 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:36257
I20260812 07:09:48.413715 32495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:36257 every 8 connection(s)
I20260812 07:09:48.415169 32421 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:09:48.419135 32496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:09:48.422696 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 32421
I20260812 07:09:48.422931 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:09:48.423818 6388 external_mini_cluster.cc:1428] Setting key 8573335c64ef263bbab9c9efbf69dead
I20260812 07:09:48.428160 32496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.61.62" port: 36257 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 07:09:48 dist-test-slave-69rg krb5kdc[32404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518588, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.61.61@KRBTEST.COM; defaulting to no policy
I20260812 07:09:48.462028 32498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:48.432376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58608 (local address 127.6.61.62:36257)
0812 07:09:48.433321 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:48.433338 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:48.438708 (+ 5370us) server_negotiation.cc:407] Connection header received
0812 07:09:48.441766 (+ 3058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:48.441772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:48.442144 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:48.442327 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:48.444115 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:48.445425 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:48.447804 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:48.448407 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:48.448550 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:48.453307 (+ 4757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:48.453319 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:48.453335 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:48.453369 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:48.457392 (+ 4023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:48.458268 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:48.458278 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:48.458282 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:48.458487 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:48.458961 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:48.458968 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:48.458972 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:48.459758 (+ 786us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:48.460003 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:48.460547 (+ 544us) server_negotiation.cc:299] Negotiation successful
0812 07:09:48.461008 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":160,"server-negotiator.queue_time_us":681,"thread_start_us":432,"threads_started":1}
I20260812 07:09:48.462072 32500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:48.432376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:36257 (local address 127.0.0.1:58608)
0812 07:09:48.438476 (+ 6100us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:48.438533 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 07:09:48.441259 (+ 2726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:09:48.442446 (+ 1187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:09:48.442458 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:09:48.443335 (+ 877us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:09:48.443856 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:48.443887 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:48.445601 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:48.445610 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:48.447629 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:48.447647 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:48.448691 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:48.448702 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:48.448928 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:48.449734 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:09:48.449776 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:09:48.453033 (+ 3257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:09:48.457572 (+ 4539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:48.457583 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:48.457594 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:48.458107 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:48.458612 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:48.458615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:48.458618 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:48.458807 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:48.459909 (+ 1102us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:09:48.459918 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:09:48.460109 (+ 191us) client_negotiation.cc:769] Sending connection context
0812 07:09:48.460369 (+ 260us) client_negotiation.cc:240] Negotiation successful
0812 07:09:48.460855 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5809,"spinlock_wait_cycles":16768,"thread_start_us":5573,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.61@KRBTEST.COM" created.
I20260812 07:09:48.465479 32496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.61.61" port: 34701 } has no permanent_uuid. Determining permanent_uuid...
I20260812 07:09:48.466648 32500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:48.466249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:34701 (local address 127.0.0.1:36876)
0812 07:09:48.466423 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:48.466533 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.61:34701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260812 07:09:48.466912 32440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.61.61:34701: connect: Connection refused (error 111)
I20260812 07:09:48.469040 32500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:48.468303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:34701 (local address 127.0.0.1:36890)
0812 07:09:48.468585 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:48.468890 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.61:34701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
W20260812 07:09:48.469292 32496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.61:34701: Network error: Client connection negotiation failed: client connection to 127.6.61.61:34701: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.61.keytab.
Entry for principal kudu/127.6.61.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.61.keytab.
I20260812 07:09:48.505252 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.61:34701
--webserver_interface=127.6.61.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.61.keytab
--principal=kudu/127.6.61.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:46551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:36257,127.6.61.61:34701,127.6.61.60:42729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:09:48.518678 32496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.61" port: 34701 } attempt: 1
I20260812 07:09:48.520198 32500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:48.519604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:34701 (local address 127.0.0.1:36904)
0812 07:09:48.519862 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:48.519981 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.61:34701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260812 07:09:48.521461 32500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:48.520990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:34701 (local address 127.0.0.1:36918)
0812 07:09:48.521177 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:48.521281 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.61:34701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260812 07:09:48.521735 32496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.61:34701: Network error: Client connection negotiation failed: client connection to 127.6.61.61:34701: connect: Connection refused (error 111)
I20260812 07:09:48.590093 32496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.61" port: 34701 } attempt: 2
I20260812 07:09:48.591495 32500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:48.590951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:34701 (local address 127.0.0.1:36928)
0812 07:09:48.591219 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:48.591344 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.61:34701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260812 07:09:48.592722 32500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:48.592342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:34701 (local address 127.0.0.1:36940)
0812 07:09:48.592497 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:48.592588 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.61:34701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260812 07:09:48.592983 32496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.61:34701: Network error: Client connection negotiation failed: client connection to 127.6.61.61:34701: connect: Connection refused (error 111)
I20260812 07:09:48.684260 32496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.61" port: 34701 } attempt: 3
I20260812 07:09:48.685670 32500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:48.685066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:34701 (local address 127.0.0.1:36942)
0812 07:09:48.685304 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:48.685509 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.61:34701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260812 07:09:48.686759 32500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:48.686390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:34701 (local address 127.0.0.1:36950)
0812 07:09:48.686538 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:48.686623 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.61:34701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 07:09:48.686981 32496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.61:34701: Network error: Client connection negotiation failed: client connection to 127.6.61.61:34701: connect: Connection refused (error 111)
I20260812 07:09:48.830303 32496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.61" port: 34701 } attempt: 4
I20260812 07:09:48.831710 32500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:48.831125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:34701 (local address 127.0.0.1:36960)
0812 07:09:48.831470 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:48.831561 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.61:34701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
I20260812 07:09:48.832834 32500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:48.832437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:34701 (local address 127.0.0.1:36968)
0812 07:09:48.832613 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:48.832707 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.61:34701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260812 07:09:48.833112 32496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.61:34701: Network error: Client connection negotiation failed: client connection to 127.6.61.61:34701: connect: Connection refused (error 111)
W20260812 07:09:48.859272 32505 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 07:09:48.859684 32505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:09:48.859836 32505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:09:48.859995 32505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:09:48.867563 32505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:09:48.867666 32505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:09:48.867762 32505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:09:48.867843 32505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:09:48.867937 32505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:09:48.878420 32505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:46551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:36257,127.6.61.61:34701,127.6.61.60:42729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.61.keytab
--principal=kudu/127.6.61.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.61:34701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.61.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:09:48.879999 32505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:09:48.881399 32505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:09:48.894084 32510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:48.894076 32513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:48.894119 32511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:09:48.894887 32505 server_base.cc:1034] running on GCE node
I20260812 07:09:48.895799 32505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:09:48.897392 32505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:09:48.898660 32505 hybrid_clock.cc:648] HybridClock initialized: now 1786518588898593 us; error 43 us; skew 500 ppm
I20260812 07:09:48.899041 32505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:09:48 dist-test-slave-69rg krb5kdc[32404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518588, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:09:48.903250 32505 init.cc:377] Logged in from keytab as kudu/127.6.61.61@KRBTEST.COM (short username kudu)
I20260812 07:09:48.905076 32505 webserver.cc:460] Webserver started at http://127.6.61.61:35461/ using document root <none> and password file <none>
I20260812 07:09:48.905956 32505 fs_manager.cc:359] Metadata directory not provided
I20260812 07:09:48.906114 32505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:09:48.906492 32505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:09:48.909912 32505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-1/data/instance:
uuid: "8a470d4a74ac4e2c8f0b777f50f39500"
format_stamp: "Formatted at 2026-08-12 07:09:48 on dist-test-slave-69rg"
server_key: "8ab0fcfe7f246638f0ffcd089890a8e7"
server_key_iv: "3fa26114d7d22082d3f0b8aa7c4df8dd"
server_key_version: "encryptionkey@0"
I20260812 07:09:48.910816 32505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-1/wal/instance:
uuid: "8a470d4a74ac4e2c8f0b777f50f39500"
format_stamp: "Formatted at 2026-08-12 07:09:48 on dist-test-slave-69rg"
server_key: "8ab0fcfe7f246638f0ffcd089890a8e7"
server_key_iv: "3fa26114d7d22082d3f0b8aa7c4df8dd"
server_key_version: "encryptionkey@0"
I20260812 07:09:48.917236 32505 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 07:09:48.921007 32522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:09:48.922660 32505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:09:48.922878 32505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-1/wal
uuid: "8a470d4a74ac4e2c8f0b777f50f39500"
format_stamp: "Formatted at 2026-08-12 07:09:48 on dist-test-slave-69rg"
server_key: "8ab0fcfe7f246638f0ffcd089890a8e7"
server_key_iv: "3fa26114d7d22082d3f0b8aa7c4df8dd"
server_key_version: "encryptionkey@0"
I20260812 07:09:48.923132 32505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:09:48.939368 32505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:09:48.943972 32505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:09:48.944286 32505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:09:48.964962 32505 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.61:34701
I20260812 07:09:48.965047 32584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.61:34701 every 8 connection(s)
I20260812 07:09:48.966486 32505 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-1/data/info.pb
I20260812 07:09:48.970407 32585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:09:48.971593 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 32505
I20260812 07:09:48.971815 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-1/wal/instance
I20260812 07:09:48.972307 6388 external_mini_cluster.cc:1428] Setting key a09ad6d4550e4c12dad5e722b2ba82cd
I20260812 07:09:48.977352 32585 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.61.62" port: 36257 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 07:09:48 dist-test-slave-69rg krb5kdc[32404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518588, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.61@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:09:48.998703 32587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:48.980202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58620 (local address 127.6.61.62:36257)
0812 07:09:48.980795 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:48.980800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:48.982018 (+ 1218us) server_negotiation.cc:407] Connection header received
0812 07:09:48.983969 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:48.983974 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:48.984071 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:48.984198 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:48.985873 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:48.986624 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:48.988177 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:48.988592 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:48.988731 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:48.992423 (+ 3692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:48.992429 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:48.992432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:48.992477 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:48.995383 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:48.996167 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:48.996174 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:48.996177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:48.996312 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:48.996761 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:48.996766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:48.996769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:48.997058 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:48.997241 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:48.998038 (+ 797us) server_negotiation.cc:299] Negotiation successful
0812 07:09:48.998295 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":340,"threads_started":1}
I20260812 07:09:48.999723 32588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:48.980834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:36257 (local address 127.0.0.1:58620)
0812 07:09:48.981800 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:48.981893 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0812 07:09:48.983672 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:09:48.984465 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:09:48.984474 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:09:48.985022 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:09:48.985597 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:48.985615 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:48.986806 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:48.986810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:48.988017 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:48.988030 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:48.988847 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:48.988856 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:48.989089 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:48.989855 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:09:48.989898 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:09:48.992159 (+ 2261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:09:48.995539 (+ 3380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:48.995549 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:48.995567 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:48.996034 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:48.996439 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:48.996443 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:48.996447 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:48.996640 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:48.997216 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:09:48.997224 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:09:48.997598 (+ 374us) client_negotiation.cc:769] Sending connection context
0812 07:09:48.997943 (+ 345us) client_negotiation.cc:240] Negotiation successful
0812 07:09:48.998514 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":376,"threads_started":1}
I20260812 07:09:49.002074 32585 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.61.61" port: 34701 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.61.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.60@KRBTEST.COM" created.
Aug 12 07:09:49 dist-test-slave-69rg krb5kdc[32404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518588, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.61@KRBTEST.COM for kudu/127.6.61.61@KRBTEST.COM
I20260812 07:09:49.017563 32588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.002852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:34701 (local address 127.0.0.1:36982)
0812 07:09:49.003035 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:49.003051 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:09:49.003150 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:09:49.004130 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:09:49.004135 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:09:49.004587 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:09:49.004777 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:49.004786 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.006398 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.006406 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:49.007312 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:49.007324 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.008529 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.008535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:49.008723 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:49.009403 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:09:49.009421 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:09:49.011620 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:09:49.014995 (+ 3375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:49.015000 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:49.015005 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:49.015374 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:49.015764 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:49.015770 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:49.015773 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:49.015918 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:49.016528 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:09:49.016532 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:09:49.016697 (+ 165us) client_negotiation.cc:769] Sending connection context
0812 07:09:49.016841 (+ 144us) client_negotiation.cc:240] Negotiation successful
0812 07:09:49.017132 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260812 07:09:49.017611 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.003143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36982 (local address 127.6.61.61:34701)
0812 07:09:49.003696 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:49.003701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:49.003727 (+ 26us) server_negotiation.cc:407] Connection header received
0812 07:09:49.003783 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:49.003789 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:49.003977 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:49.004152 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:49.004956 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.006214 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.007501 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.008361 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.008528 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:49.011855 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:49.011867 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:49.011883 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:49.011912 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:49.014846 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:49.015523 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:49.015530 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:49.015532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:49.015652 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:49.016025 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:49.016033 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:49.016036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:49.016397 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:49.016627 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:49.017004 (+ 377us) server_negotiation.cc:299] Negotiation successful
0812 07:09:49.017224 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":263,"threads_started":1}
I20260812 07:09:49.020466 32585 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.61.60" port: 42729 } has no permanent_uuid. Determining permanent_uuid...
I20260812 07:09:49.021762 32588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.021296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:42729 (local address 127.0.0.1:58792)
0812 07:09:49.021492 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:49.021605 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260812 07:09:49.022140 32523 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.61.60:42729: connect: Connection refused (error 111)
I20260812 07:09:49.023689 32588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.023377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:42729 (local address 127.0.0.1:58798)
0812 07:09:49.023517 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:49.023583 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 07:09:49.023942 32585 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.60:42729: Network error: Client connection negotiation failed: client connection to 127.6.61.60:42729: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.60.keytab.
Entry for principal kudu/127.6.61.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.60.keytab.
I20260812 07:09:49.054955 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.60:42729
--webserver_interface=127.6.61.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.60.keytab
--principal=kudu/127.6.61.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:46551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:36257,127.6.61.61:34701,127.6.61.60:42729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:09:49.073288 32585 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.60" port: 42729 } attempt: 1
I20260812 07:09:49.074736 32588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.074202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:42729 (local address 127.0.0.1:58808)
0812 07:09:49.074441 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:49.074546 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260812 07:09:49.075840 32588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.075432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:42729 (local address 127.0.0.1:58810)
0812 07:09:49.075585 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:49.075677 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 07:09:49.076059 32585 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.60:42729: Network error: Client connection negotiation failed: client connection to 127.6.61.60:42729: connect: Connection refused (error 111)
I20260812 07:09:49.132448 32496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.61" port: 34701 } attempt: 5
Aug 12 07:09:49 dist-test-slave-69rg krb5kdc[32404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518588, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for kudu/127.6.61.61@KRBTEST.COM
I20260812 07:09:49.144317 32585 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.60" port: 42729 } attempt: 2
I20260812 07:09:49.145640 32588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.145066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:42729 (local address 127.0.0.1:58826)
0812 07:09:49.145314 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:49.145433 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260812 07:09:49.147243 32588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.146352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:42729 (local address 127.0.0.1:58836)
0812 07:09:49.146981 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:49.147063 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511}
W20260812 07:09:49.147486 32585 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.60:42729: Network error: Client connection negotiation failed: client connection to 127.6.61.60:42729: connect: Connection refused (error 111)
I20260812 07:09:49.149127 32500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.133360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:34701 (local address 127.0.0.1:36992)
0812 07:09:49.133611 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:49.133635 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 07:09:49.133801 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:09:49.134227 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:09:49.134235 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:09:49.134850 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:09:49.135120 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:49.135137 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.136222 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.136230 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:49.137848 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:49.137867 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.138671 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.138678 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:49.138882 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:49.139570 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:09:49.139595 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:09:49.142297 (+ 2702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:09:49.146187 (+ 3890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:49.146193 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:49.146197 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:49.146731 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:49.147198 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:49.147206 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:49.147211 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:49.147427 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:49.148290 (+ 863us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:09:49.148294 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:09:49.148456 (+ 162us) client_negotiation.cc:769] Sending connection context
0812 07:09:49.148614 (+ 158us) client_negotiation.cc:240] Negotiation successful
0812 07:09:49.148884 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260812 07:09:49.149526 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.133590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36992 (local address 127.6.61.61:34701)
0812 07:09:49.133779 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:49.133784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:49.133804 (+ 20us) server_negotiation.cc:407] Connection header received
0812 07:09:49.133972 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:49.133976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:49.134061 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:49.134147 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:49.135309 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.136049 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.138008 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.138504 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.138616 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:49.142535 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:49.142540 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:49.142544 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:49.142582 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:49.146002 (+ 3420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:49.146881 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:49.146887 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:49.146890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:49.147048 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:49.147600 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:49.147607 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:49.147612 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:49.148107 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:49.148301 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:49.148808 (+ 507us) server_negotiation.cc:299] Negotiation successful
0812 07:09:49.149117 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 07:09:49.150633 32496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.61.60" port: 42729 } has no permanent_uuid. Determining permanent_uuid...
I20260812 07:09:49.151787 32500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.151334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:42729 (local address 127.0.0.1:58846)
0812 07:09:49.151531 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:49.151631 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260812 07:09:49.153115 32500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.152659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:42729 (local address 127.0.0.1:58860)
0812 07:09:49.152844 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:49.152935 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260812 07:09:49.153367 32496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.60:42729: Network error: Client connection negotiation failed: client connection to 127.6.61.60:42729: connect: Connection refused (error 111)
I20260812 07:09:49.204777 32496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.60" port: 42729 } attempt: 1
I20260812 07:09:49.206125 32500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.205528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:42729 (local address 127.0.0.1:53238)
0812 07:09:49.205741 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:49.205930 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260812 07:09:49.207295 32500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.206893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:42729 (local address 127.0.0.1:53250)
0812 07:09:49.207081 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:49.207174 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260812 07:09:49.207639 32496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.60:42729: Network error: Client connection negotiation failed: client connection to 127.6.61.60:42729: connect: Connection refused (error 111)
I20260812 07:09:49.238770 32585 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.60" port: 42729 } attempt: 3
I20260812 07:09:49.240291 32588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.239568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:42729 (local address 127.0.0.1:53262)
0812 07:09:49.239893 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:49.240042 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260812 07:09:49.241901 32588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.241199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:42729 (local address 127.0.0.1:53272)
0812 07:09:49.241544 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:49.241664 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192}
W20260812 07:09:49.242192 32585 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.60:42729: Network error: Client connection negotiation failed: client connection to 127.6.61.60:42729: connect: Connection refused (error 111)
I20260812 07:09:49.275934 32496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.60" port: 42729 } attempt: 2
I20260812 07:09:49.277367 32500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.276751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:42729 (local address 127.0.0.1:53276)
0812 07:09:49.277056 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:49.277149 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260812 07:09:49.278550 32500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.278166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:42729 (local address 127.0.0.1:53280)
0812 07:09:49.278350 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:49.278433 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260812 07:09:49.278793 32496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.60:42729: Network error: Client connection negotiation failed: client connection to 127.6.61.60:42729: connect: Connection refused (error 111)
I20260812 07:09:49.385035 32496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.60" port: 42729 } attempt: 3
I20260812 07:09:49.385435 32585 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.60" port: 42729 } attempt: 4
I20260812 07:09:49.386451 32500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.385800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:42729 (local address 127.0.0.1:53286)
0812 07:09:49.386109 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:49.386237 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260812 07:09:49.386927 32588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.386150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:42729 (local address 127.0.0.1:53298)
0812 07:09:49.386377 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:49.386487 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260812 07:09:49.387724 32500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.387200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:42729 (local address 127.0.0.1:53314)
0812 07:09:49.387469 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:49.387570 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260812 07:09:49.388001 32496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.60:42729: Network error: Client connection negotiation failed: client connection to 127.6.61.60:42729: connect: Connection refused (error 111)
I20260812 07:09:49.388129 32588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.387641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:42729 (local address 127.0.0.1:53318)
0812 07:09:49.387890 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:49.387987 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260812 07:09:49.388350 32585 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.60:42729: Network error: Client connection negotiation failed: client connection to 127.6.61.60:42729: connect: Connection refused (error 111)
W20260812 07:09:49.432304 32593 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 07:09:49.432659 32593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:09:49.432726 32593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:09:49.432801 32593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:09:49.440302 32593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:09:49.440407 32593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:09:49.440464 32593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:09:49.440508 32593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:09:49.440577 32593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:09:49.450932 32593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:46551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:36257,127.6.61.61:34701,127.6.61.60:42729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.60.keytab
--principal=kudu/127.6.61.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.60:42729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.61.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:09:49.452256 32593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:09:49.453552 32593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:09:49.465943 32601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:49.465976 32599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:49.466065 32603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:09:49.467976 32593 server_base.cc:1034] running on GCE node
I20260812 07:09:49.468885 32593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:09:49.470541 32593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:09:49.471827 32593 hybrid_clock.cc:648] HybridClock initialized: now 1786518589471736 us; error 63 us; skew 500 ppm
I20260812 07:09:49.472235 32593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:09:49 dist-test-slave-69rg krb5kdc[32404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518589, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:09:49.475795 32593 init.cc:377] Logged in from keytab as kudu/127.6.61.60@KRBTEST.COM (short username kudu)
I20260812 07:09:49.477555 32593 webserver.cc:460] Webserver started at http://127.6.61.60:42587/ using document root <none> and password file <none>
I20260812 07:09:49.478372 32593 fs_manager.cc:359] Metadata directory not provided
I20260812 07:09:49.478524 32593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:09:49.478876 32593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:09:49.482060 32593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-2/data/instance:
uuid: "53071e25f9434e8ca88182548146bd7e"
format_stamp: "Formatted at 2026-08-12 07:09:49 on dist-test-slave-69rg"
server_key: "736d1d809850b6253c0d227462b80c53"
server_key_iv: "63e5156e724ffff19d65016c217dccca"
server_key_version: "encryptionkey@0"
I20260812 07:09:49.482884 32593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-2/wal/instance:
uuid: "53071e25f9434e8ca88182548146bd7e"
format_stamp: "Formatted at 2026-08-12 07:09:49 on dist-test-slave-69rg"
server_key: "736d1d809850b6253c0d227462b80c53"
server_key_iv: "63e5156e724ffff19d65016c217dccca"
server_key_version: "encryptionkey@0"
I20260812 07:09:49.488348 32593 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260812 07:09:49.492388 32610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:09:49.494058 32593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:09:49.494273 32593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-2/wal
uuid: "53071e25f9434e8ca88182548146bd7e"
format_stamp: "Formatted at 2026-08-12 07:09:49 on dist-test-slave-69rg"
server_key: "736d1d809850b6253c0d227462b80c53"
server_key_iv: "63e5156e724ffff19d65016c217dccca"
server_key_version: "encryptionkey@0"
I20260812 07:09:49.494524 32593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:09:49.509649 32593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:09:49.514896 32593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:09:49.515232 32593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:09:49.535344 32593 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.60:42729
I20260812 07:09:49.535374 32668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.60:42729 every 8 connection(s)
I20260812 07:09:49.536808 32593 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-2/data/info.pb
I20260812 07:09:49.540879 32669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:09:49.542446 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 32593
I20260812 07:09:49.542804 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518208686283-6388-0/minicluster-data/master-2/wal/instance
I20260812 07:09:49.543361 6388 external_mini_cluster.cc:1428] Setting key 594737aab27a9c0f1627085e48922679
I20260812 07:09:49.550951 32669 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.61.62" port: 36257 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 07:09:49 dist-test-slave-69rg krb5kdc[32404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.60@KRBTEST.COM
Aug 12 07:09:49 dist-test-slave-69rg krb5kdc[32404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518589, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.60@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:09:49.565281 32496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.60" port: 42729 } attempt: 4
I20260812 07:09:49.569708 32672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.545567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53324 (local address 127.6.61.60:42729)
0812 07:09:49.546522 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:49.546534 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:49.546584 (+ 50us) server_negotiation.cc:407] Connection header received
0812 07:09:49.548187 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:49.548211 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:49.548648 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:49.548969 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:49.551046 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.552552 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.554429 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.555476 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.555682 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:49.560459 (+ 4777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:49.560465 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:49.560476 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:49.560512 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:49.563779 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:49.564556 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:49.564563 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:49.564571 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:49.564726 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:49.565292 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:49.565299 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:49.565305 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:49.566346 (+ 1041us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:49.567394 (+ 1048us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:49.567744 (+ 350us) server_negotiation.cc:299] Negotiation successful
0812 07:09:49.568170 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":459,"threads_started":1}
I20260812 07:09:49.571686 32674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.553914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:36257 (local address 127.0.0.1:55768)
0812 07:09:49.554948 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:49.555008 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 07:09:49.555175 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:09:49.556023 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:09:49.556032 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:09:49.556935 (+ 903us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:09:49.557308 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:49.557325 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.558559 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.558565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:49.559737 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:49.559756 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.560719 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.560726 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:49.560939 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:49.561638 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:09:49.561656 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:09:49.564191 (+ 2535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:09:49.568523 (+ 4332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:49.568534 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:49.568541 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:49.569154 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:49.569596 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:49.569601 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:49.569605 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:49.569784 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:49.570485 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:09:49.570493 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:09:49.570678 (+ 185us) client_negotiation.cc:769] Sending connection context
0812 07:09:49.570852 (+ 174us) client_negotiation.cc:240] Negotiation successful
0812 07:09:49.571213 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"thread_start_us":621,"threads_started":1}
I20260812 07:09:49.572551 32673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.553909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55768 (local address 127.6.61.62:36257)
0812 07:09:49.554617 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:49.554622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:49.555113 (+ 491us) server_negotiation.cc:407] Connection header received
0812 07:09:49.555370 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:49.555375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:49.555459 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:49.555562 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:49.557499 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.558385 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.559912 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.560505 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.560677 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:49.564409 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:49.564415 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:49.564418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:49.564453 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:49.568378 (+ 3925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:49.569294 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:49.569300 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:49.569303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:49.569467 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:49.569964 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:49.569971 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:49.569976 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:49.570340 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:49.570508 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:49.570995 (+ 487us) server_negotiation.cc:299] Negotiation successful
0812 07:09:49.571214 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":327,"threads_started":1}
I20260812 07:09:49.574304 32669 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.61.61" port: 34701 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 07:09:49 dist-test-slave-69rg krb5kdc[32404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518588, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for kudu/127.6.61.60@KRBTEST.COM
Aug 12 07:09:49 dist-test-slave-69rg krb5kdc[32404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
Aug 12 07:09:49 dist-test-slave-69rg krb5kdc[32404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518589, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.60@KRBTEST.COM for kudu/127.6.61.61@KRBTEST.COM
I20260812 07:09:49.583678 32500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.566081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:42729 (local address 127.0.0.1:53340)
0812 07:09:49.566466 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:49.566492 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 07:09:49.566628 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:09:49.567784 (+ 1156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:09:49.567790 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:09:49.568371 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:09:49.568589 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:49.568599 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.570122 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.570131 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:49.571529 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:49.571542 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.572552 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.572558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:49.572775 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:49.573478 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:09:49.573500 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:09:49.576497 (+ 2997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:09:49.580606 (+ 4109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:49.580614 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:49.580621 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:49.581195 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:49.581623 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:49.581630 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:49.581635 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:49.581877 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:49.582591 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:09:49.582608 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:09:49.582804 (+ 196us) client_negotiation.cc:769] Sending connection context
0812 07:09:49.582963 (+ 159us) client_negotiation.cc:240] Negotiation successful
0812 07:09:49.583258 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212}
I20260812 07:09:49.583993 32675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.566109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53340 (local address 127.6.61.60:42729)
0812 07:09:49.567358 (+ 1249us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:49.567363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:49.567385 (+ 22us) server_negotiation.cc:407] Connection header received
0812 07:09:49.567434 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:49.567438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:49.567519 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:49.567712 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:49.568758 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.569974 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.571693 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.572319 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.572516 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:49.576731 (+ 4215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:49.576738 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:49.576743 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:49.576779 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:49.580466 (+ 3687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:49.581333 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:49.581338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:49.581341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:49.581487 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:49.581985 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:49.581990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:49.581993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:49.582360 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:49.582514 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:49.583062 (+ 548us) server_negotiation.cc:299] Negotiation successful
0812 07:09:49.583269 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1107,"thread_start_us":899,"threads_started":1}
I20260812 07:09:49.590380 32673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.573796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55776 (local address 127.6.61.62:36257)
0812 07:09:49.573995 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:49.574000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:49.574018 (+ 18us) server_negotiation.cc:407] Connection header received
0812 07:09:49.574201 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:49.574207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:49.574305 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:49.575424 (+ 1119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:49.576198 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.577259 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.578640 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.579194 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.579335 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:49.584252 (+ 4917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:49.584259 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:49.584264 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:49.584308 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:49.587251 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:49.587945 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:49.587950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:49.587954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:49.588114 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:49.588560 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:49.588565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:49.588570 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:49.588945 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:49.589258 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:49.589617 (+ 359us) server_negotiation.cc:299] Negotiation successful
0812 07:09:49.589900 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260812 07:09:49.590723 32674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.574955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:34701 (local address 127.0.0.1:50550)
0812 07:09:49.575187 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:49.575205 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 07:09:49.575294 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:09:49.575729 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:09:49.575734 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:09:49.576154 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:09:49.576337 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:49.576349 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.577612 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.577619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:49.579211 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:49.579228 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.580502 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.580511 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:49.580711 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:49.581249 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:09:49.581294 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:09:49.584164 (+ 2870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:09:49.587891 (+ 3727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:49.587899 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:49.587902 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:49.588385 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:49.588836 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:49.588841 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:49.588845 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:49.589014 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:49.589751 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:09:49.589758 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:09:49.589940 (+ 182us) client_negotiation.cc:769] Sending connection context
0812 07:09:49.590100 (+ 160us) client_negotiation.cc:240] Negotiation successful
0812 07:09:49.590385 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260812 07:09:49.590914 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.575217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50550 (local address 127.6.61.61:34701)
0812 07:09:49.575428 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:49.575434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:49.575450 (+ 16us) server_negotiation.cc:407] Connection header received
0812 07:09:49.575506 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:49.575512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:49.575592 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:49.575708 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:49.576504 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.577433 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.579410 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.580109 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.580221 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:49.584338 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:49.584344 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:49.584349 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:49.584393 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:49.587742 (+ 3349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:49.588518 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:49.588523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:49.588526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:49.588705 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:49.589158 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:49.589163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:49.589166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:49.589508 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:49.589649 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:49.590294 (+ 645us) server_negotiation.cc:299] Negotiation successful
0812 07:09:49.590538 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260812 07:09:49.591681 32496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bf80683ae714a0cacf072bd357cdaea: Bootstrap starting.
I20260812 07:09:49.591948 32669 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.61.60" port: 42729 } has no permanent_uuid. Determining permanent_uuid...
I20260812 07:09:49.595741 32496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bf80683ae714a0cacf072bd357cdaea: Neither blocks nor log segments found. Creating new log.
I20260812 07:09:49.597069 32496 log.cc:824] T 00000000000000000000000000000000 P 3bf80683ae714a0cacf072bd357cdaea: Log is configured to *not* fsync() on all Append() calls
Aug 12 07:09:49 dist-test-slave-69rg krb5kdc[32404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518589, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.60@KRBTEST.COM for kudu/127.6.61.60@KRBTEST.COM
I20260812 07:09:49.600805 32496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bf80683ae714a0cacf072bd357cdaea: No bootstrap required, opened a new log
Aug 12 07:09:49 dist-test-slave-69rg krb5kdc[32404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.61@KRBTEST.COM
I20260812 07:09:49.606622 32674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.592684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:42729 (local address 127.0.0.1:53352)
0812 07:09:49.593185 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:49.593200 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:09:49.593350 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:09:49.593661 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:09:49.593664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:09:49.594111 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:09:49.594247 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:49.594255 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.595313 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.595319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:49.596496 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:49.596507 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.597201 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.597207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:49.597409 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:49.598164 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:09:49.598191 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:09:49.600603 (+ 2412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:09:49.604149 (+ 3546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:49.604154 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:49.604157 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:49.604519 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:49.604893 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:49.604899 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:49.604903 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:49.605058 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:49.605595 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:09:49.605598 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:09:49.605686 (+ 88us) client_negotiation.cc:769] Sending connection context
0812 07:09:49.605821 (+ 135us) client_negotiation.cc:240] Negotiation successful
0812 07:09:49.606204 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427}
I20260812 07:09:49.606683 32675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.592940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53352 (local address 127.6.61.60:42729)
0812 07:09:49.593096 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:49.593101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:49.593438 (+ 337us) server_negotiation.cc:407] Connection header received
0812 07:09:49.593480 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:49.593483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:49.593541 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:49.593641 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:49.594379 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.595160 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.596632 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.597040 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.597164 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:49.600793 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:49.600798 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:49.600801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:49.600828 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:49.603997 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:49.604640 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:49.604648 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:49.604651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:49.604786 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:49.605176 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:49.605181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:49.605186 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:49.605449 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:49.605537 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:49.605995 (+ 458us) server_negotiation.cc:299] Negotiation successful
0812 07:09:49.606343 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":143,"server-negotiator.queue_time_us":46}
I20260812 07:09:49.610925 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.592893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50558 (local address 127.6.61.61:34701)
0812 07:09:49.593049 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:49.593055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:49.593072 (+ 17us) server_negotiation.cc:407] Connection header received
0812 07:09:49.593122 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:49.593127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:49.593197 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:49.593297 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:49.594995 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.596070 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.597930 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.598372 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.598521 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:49.604254 (+ 5733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:49.604262 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:49.604269 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:49.604311 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:49.607912 (+ 3601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:49.608769 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:49.608775 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:49.608779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:49.608945 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:49.609418 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:49.609424 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:49.609428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:49.609785 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:49.610081 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:49.610353 (+ 272us) server_negotiation.cc:299] Negotiation successful
0812 07:09:49.610577 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 07:09:49.611152 32496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bf80683ae714a0cacf072bd357cdaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bf80683ae714a0cacf072bd357cdaea" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 36257 } } peers { permanent_uuid: "8a470d4a74ac4e2c8f0b777f50f39500" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 34701 } } peers { permanent_uuid: "53071e25f9434e8ca88182548146bd7e" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 42729 } }
I20260812 07:09:49.611569 32496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bf80683ae714a0cacf072bd357cdaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bf80683ae714a0cacf072bd357cdaea, State: Initialized, Role: FOLLOWER
I20260812 07:09:49.612119 32669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53071e25f9434e8ca88182548146bd7e: Bootstrap starting.
I20260812 07:09:49.612023 32496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bf80683ae714a0cacf072bd357cdaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bf80683ae714a0cacf072bd357cdaea" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 36257 } } peers { permanent_uuid: "8a470d4a74ac4e2c8f0b777f50f39500" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 34701 } } peers { permanent_uuid: "53071e25f9434e8ca88182548146bd7e" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 42729 } }
I20260812 07:09:49.613970 32677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bf80683ae714a0cacf072bd357cdaea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bf80683ae714a0cacf072bd357cdaea" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 36257 } } peers { permanent_uuid: "8a470d4a74ac4e2c8f0b777f50f39500" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 34701 } } peers { permanent_uuid: "53071e25f9434e8ca88182548146bd7e" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 42729 } } }
I20260812 07:09:49.614444 32677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bf80683ae714a0cacf072bd357cdaea [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:09:49.615121 32496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bf80683ae714a0cacf072bd357cdaea [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:09:49.616128 32669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53071e25f9434e8ca88182548146bd7e: Neither blocks nor log segments found. Creating new log.
I20260812 07:09:49.616577 6388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:09:49.617439 32669 log.cc:824] T 00000000000000000000000000000000 P 53071e25f9434e8ca88182548146bd7e: Log is configured to *not* fsync() on all Append() calls
I20260812 07:09:49.620945 32468 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3bf80683ae714a0cacf072bd357cdaea"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55776
I20260812 07:09:49.622061 32468 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bf80683ae714a0cacf072bd357cdaea [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 07:09:49.622268 32468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bf80683ae714a0cacf072bd357cdaea [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:09:49.624462 32669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53071e25f9434e8ca88182548146bd7e: No bootstrap required, opened a new log
I20260812 07:09:49.625558 32468 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bf80683ae714a0cacf072bd357cdaea [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bf80683ae714a0cacf072bd357cdaea" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 36257 } } peers { permanent_uuid: "8a470d4a74ac4e2c8f0b777f50f39500" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 34701 } } peers { permanent_uuid: "53071e25f9434e8ca88182548146bd7e" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 42729 } }
W20260812 07:09:49.626507 32694 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3bf80683ae714a0cacf072bd357cdaea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 07:09:49.626644 32694 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 07:09:49.627138 32468 leader_election.cc:288] T 00000000000000000000000000000000 P 3bf80683ae714a0cacf072bd357cdaea [CANDIDATE]: Term 1 election: Requested vote from peers 8a470d4a74ac4e2c8f0b777f50f39500 (127.6.61.61:34701), 53071e25f9434e8ca88182548146bd7e (127.6.61.60:42729)
I20260812 07:09:49.637794 32669 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53071e25f9434e8ca88182548146bd7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bf80683ae714a0cacf072bd357cdaea" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 36257 } } peers { permanent_uuid: "8a470d4a74ac4e2c8f0b777f50f39500" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 34701 } } peers { permanent_uuid: "53071e25f9434e8ca88182548146bd7e" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 42729 } }
I20260812 07:09:49.638410 32669 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53071e25f9434e8ca88182548146bd7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53071e25f9434e8ca88182548146bd7e, State: Initialized, Role: FOLLOWER
I20260812 07:09:49.639079 32669 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53071e25f9434e8ca88182548146bd7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bf80683ae714a0cacf072bd357cdaea" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 36257 } } peers { permanent_uuid: "8a470d4a74ac4e2c8f0b777f50f39500" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 34701 } } peers { permanent_uuid: "53071e25f9434e8ca88182548146bd7e" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 42729 } }
W20260812 07:09:49.647125 32444 tablet.cc:2369] T 00000000000000000000000000000000 P 3bf80683ae714a0cacf072bd357cdaea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:09:49.631999 32421 master_runner.cc:186] Error getting master registration for 127.6.61.61:34701: OK, instance_id { permanent_uuid: "8a470d4a74ac4e2c8f0b777f50f39500" instance_seqno: 1786518588959181 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 07:09:49.648851 32421 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 07:09:49.649909 32697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53071e25f9434e8ca88182548146bd7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bf80683ae714a0cacf072bd357cdaea" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 36257 } } peers { permanent_uuid: "8a470d4a74ac4e2c8f0b777f50f39500" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 34701 } } peers { permanent_uuid: "53071e25f9434e8ca88182548146bd7e" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 42729 } } }
I20260812 07:09:49.650657 32697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53071e25f9434e8ca88182548146bd7e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:09:49.652431 32669 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53071e25f9434e8ca88182548146bd7e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:09:49.652526 32642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3bf80683ae714a0cacf072bd357cdaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "53071e25f9434e8ca88182548146bd7e"
I20260812 07:09:49.653131 32642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53071e25f9434e8ca88182548146bd7e [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:09:49.656925 32642 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 53071e25f9434e8ca88182548146bd7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bf80683ae714a0cacf072bd357cdaea in term 1.
I20260812 07:09:49.657764 32438 leader_election.cc:302] T 00000000000000000000000000000000 P 3bf80683ae714a0cacf072bd357cdaea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bf80683ae714a0cacf072bd357cdaea, 53071e25f9434e8ca88182548146bd7e; no voters:
I20260812 07:09:49.658387 32677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bf80683ae714a0cacf072bd357cdaea [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:09:49.670197 32677 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bf80683ae714a0cacf072bd357cdaea [term 1 LEADER]: Becoming Leader. State: Replica: 3bf80683ae714a0cacf072bd357cdaea, State: Running, Role: LEADER
I20260812 07:09:49.671316 32677 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bf80683ae714a0cacf072bd357cdaea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bf80683ae714a0cacf072bd357cdaea" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 36257 } } peers { permanent_uuid: "8a470d4a74ac4e2c8f0b777f50f39500" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 34701 } } peers { permanent_uuid: "53071e25f9434e8ca88182548146bd7e" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 42729 } }
I20260812 07:09:49.687541 32585 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.60" port: 42729 } attempt: 5
I20260812 07:09:49.693809 32705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bf80683ae714a0cacf072bd357cdaea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bf80683ae714a0cacf072bd357cdaea. Latest consensus state: current_term: 1 leader_uuid: "3bf80683ae714a0cacf072bd357cdaea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bf80683ae714a0cacf072bd357cdaea" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 36257 } } peers { permanent_uuid: "8a470d4a74ac4e2c8f0b777f50f39500" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 34701 } } peers { permanent_uuid: "53071e25f9434e8ca88182548146bd7e" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 42729 } } }
I20260812 07:09:49.694326 32705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bf80683ae714a0cacf072bd357cdaea [sys.catalog]: This master's current role is: LEADER
Aug 12 07:09:49 dist-test-slave-69rg krb5kdc[32404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518588, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.61@KRBTEST.COM for kudu/127.6.61.60@KRBTEST.COM
I20260812 07:09:49.709496 32710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:09:49.709968 32672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.688932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53370 (local address 127.6.61.60:42729)
0812 07:09:49.689769 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:49.689778 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:49.689802 (+ 24us) server_negotiation.cc:407] Connection header received
0812 07:09:49.689900 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:49.689908 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:49.690010 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:49.690129 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:49.691555 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.693013 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.695472 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.696272 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.696403 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:49.701927 (+ 5524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:49.701934 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:49.701940 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:49.701982 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:49.705925 (+ 3943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:49.706982 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:49.706988 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:49.706992 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:49.707140 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:49.707658 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:49.707663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:49.707666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:49.708072 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:49.708206 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:49.709127 (+ 921us) server_negotiation.cc:299] Negotiation successful
0812 07:09:49.709436 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647}
I20260812 07:09:49.711301 32588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.688302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:42729 (local address 127.0.0.1:53370)
0812 07:09:49.688859 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:49.688884 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 07:09:49.689028 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:09:49.690392 (+ 1364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:09:49.690400 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:09:49.691063 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:09:49.691312 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:49.691332 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.693237 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.693246 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:49.695259 (+ 2013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:49.695279 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.696443 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.696449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:49.696839 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:49.697636 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:09:49.697666 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:09:49.701054 (+ 3388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:09:49.706248 (+ 5194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:49.706255 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:49.706259 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:49.706849 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:49.707302 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:49.707308 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:49.707313 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:49.707503 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:49.708328 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:09:49.708335 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:09:49.708786 (+ 451us) client_negotiation.cc:769] Sending connection context
0812 07:09:49.710535 (+ 1749us) client_negotiation.cc:240] Negotiation successful
0812 07:09:49.710890 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367}
I20260812 07:09:49.717077 32673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.654270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55786 (local address 127.6.61.62:36257)
0812 07:09:49.654521 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:49.654530 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:49.664622 (+ 10092us) server_negotiation.cc:407] Connection header received
0812 07:09:49.666685 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:49.666694 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:49.666828 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:49.666928 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:49.672784 (+ 5856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.674430 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.680191 (+ 5761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.681187 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.681328 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:49.687592 (+ 6264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:49.687601 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:49.687606 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:49.687655 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:49.692048 (+ 4393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:49.711862 (+ 19814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:49.711870 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:49.711875 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:49.712041 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:49.712553 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:49.712561 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:49.712567 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:49.713070 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:49.713356 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:49.716373 (+ 3017us) server_negotiation.cc:299] Negotiation successful
0812 07:09:49.716676 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260812 07:09:49.722697 32710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:09:49.725322 32585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a470d4a74ac4e2c8f0b777f50f39500: Bootstrap starting.
I20260812 07:09:49.733743 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.665342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50564 (local address 127.6.61.61:34701)
0812 07:09:49.665567 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:49.665576 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:49.698468 (+ 32892us) server_negotiation.cc:407] Connection header received
0812 07:09:49.698768 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:49.698774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:49.698851 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:49.698919 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:49.700882 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.702556 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.706811 (+ 4255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.707519 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.707619 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:49.718351 (+ 10732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:49.718359 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:49.718364 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:49.718418 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:49.728731 (+ 10313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:49.729931 (+ 1200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:49.729937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:49.729940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:49.730112 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:49.730603 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:49.730608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:49.730611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:49.731092 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:49.731379 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:49.732887 (+ 1508us) server_negotiation.cc:299] Negotiation successful
0812 07:09:49.733232 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 07:09:49.740325 32585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a470d4a74ac4e2c8f0b777f50f39500: Neither blocks nor log segments found. Creating new log.
I20260812 07:09:49.741885 32585 log.cc:824] T 00000000000000000000000000000000 P 8a470d4a74ac4e2c8f0b777f50f39500: Log is configured to *not* fsync() on all Append() calls
I20260812 07:09:49.743978 32675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:49.660692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53360 (local address 127.6.61.60:42729)
0812 07:09:49.660883 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:49.660891 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:49.694767 (+ 33876us) server_negotiation.cc:407] Connection header received
0812 07:09:49.695783 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:49.695791 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:49.695907 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:49.696010 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:49.698161 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.699628 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.702922 (+ 3294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:49.703974 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:49.704144 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:49.711747 (+ 7603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:49.711755 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:49.711759 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:49.711802 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:49.715641 (+ 3839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:49.716485 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:49.716490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:49.716493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:49.716675 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:49.739309 (+ 22634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:49.739320 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:49.739324 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:49.740131 (+ 807us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:49.740721 (+ 590us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:49.741953 (+ 1232us) server_negotiation.cc:299] Negotiation successful
0812 07:09:49.742484 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":66}
I20260812 07:09:49.747808 32642 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 53071e25f9434e8ca88182548146bd7e [term 1 FOLLOWER]: Refusing update from remote peer 3bf80683ae714a0cacf072bd357cdaea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:09:49.748817 32705 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3bf80683ae714a0cacf072bd357cdaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "53071e25f9434e8ca88182548146bd7e" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 42729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 07:09:49.767513 32618 tablet.cc:2369] T 00000000000000000000000000000000 P 53071e25f9434e8ca88182548146bd7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:09:49.774431 32697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53071e25f9434e8ca88182548146bd7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bf80683ae714a0cacf072bd357cdaea. Latest consensus state: current_term: 1 leader_uuid: "3bf80683ae714a0cacf072bd357cdaea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bf80683ae714a0cacf072bd357cdaea" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 36257 } } peers { permanent_uuid: "8a470d4a74ac4e2c8f0b777f50f39500" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 34701 } } peers { permanent_uuid: "53071e25f9434e8ca88182548146bd7e" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 42729 } } }
I20260812 07:09:49.774801 32697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53071e25f9434e8ca88182548146bd7e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:09:49.786640 32723 mvcc.cc:205] Tried to move back new op lower bound from 7317580143573458944 to 7317580143314259968. Current Snapshot: MvccSnapshot[applied={T|T < 7317580143573458944}]
I20260812 07:09:49.787624 32677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bf80683ae714a0cacf072bd357cdaea [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3bf80683ae714a0cacf072bd357cdaea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bf80683ae714a0cacf072bd357cdaea" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 36257 } } peers { permanent_uuid: "8a470d4a74ac4e2c8f0b777f50f39500" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 34701 } } peers { permanent_uuid: "53071e25f9434e8ca88182548146bd7e" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 42729 } } }
I20260812 07:09:49.788005 32677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bf80683ae714a0cacf072bd357cdaea [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:49.789062 32697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53071e25f9434e8ca88182548146bd7e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3bf80683ae714a0cacf072bd357cdaea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bf80683ae714a0cacf072bd357cdaea" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 36257 } } peers { permanent_uuid: "8a470d4a74ac4e2c8f0b777f50f39500" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 34701 } } peers { permanent_uuid: "53071e25f9434e8ca88182548146bd7e" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 42729 } } }
I20260812 07:09:49.789414 32697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53071e25f9434e8ca88182548146bd7e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:09:49.792327 32710 catalog_manager.cc:1347] Generated new cluster ID: 14642d29f9a04bfeb68bf99ccbc003a3
I20260812 07:09:49.792440 32710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:09:49.795616 32585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a470d4a74ac4e2c8f0b777f50f39500: No bootstrap required, opened a new log
I20260812 07:09:49.807020 32727 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 07:09:49.814368 32585 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a470d4a74ac4e2c8f0b777f50f39500 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bf80683ae714a0cacf072bd357cdaea" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 36257 } } peers { permanent_uuid: "8a470d4a74ac4e2c8f0b777f50f39500" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 34701 } } peers { permanent_uuid: "53071e25f9434e8ca88182548146bd7e" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 42729 } }
I20260812 07:09:49.814867 32585 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a470d4a74ac4e2c8f0b777f50f39500 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a470d4a74ac4e2c8f0b777f50f39500, State: Initialized, Role: FOLLOWER
I20260812 07:09:49.815393 32585 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a470d4a74ac4e2c8f0b777f50f39500 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bf80683ae714a0cacf072bd357cdaea" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 36257 } } peers { permanent_uuid: "8a470d4a74ac4e2c8f0b777f50f39500" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 34701 } } peers { permanent_uuid: "53071e25f9434e8ca88182548146bd7e" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 42729 } }
I20260812 07:09:49.827828 32585 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a470d4a74ac4e2c8f0b777f50f39500 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:09:49.813796 32593 master_runner.cc:186] Error getting master registration for 127.6.61.61:34701: OK, instance_id { permanent_uuid: "8a470d4a74ac4e2c8f0b777f50f39500" instance_seqno: 1786518588959181 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 07:09:49.828384 32554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a470d4a74ac4e2c8f0b777f50f39500 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:09:49.828480 32593 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 07:09:49.829010 32555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3bf80683ae714a0cacf072bd357cdaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8a470d4a74ac4e2c8f0b777f50f39500"
I20260812 07:09:49.822388 32728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a470d4a74ac4e2c8f0b777f50f39500 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bf80683ae714a0cacf072bd357cdaea" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 36257 } } peers { permanent_uuid: "8a470d4a74ac4e2c8f0b777f50f39500" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 34701 } } peers { permanent_uuid: "53071e25f9434e8ca88182548146bd7e" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 42729 } } }
I20260812 07:09:49.831825 32728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a470d4a74ac4e2c8f0b777f50f39500 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:09:49.832032 32710 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:09:49.834107 32710 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:09:49.834726 32554 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8a470d4a74ac4e2c8f0b777f50f39500 [term 1 FOLLOWER]: Refusing update from remote peer 3bf80683ae714a0cacf072bd357cdaea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:09:49.838390 32555 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8a470d4a74ac4e2c8f0b777f50f39500 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bf80683ae714a0cacf072bd357cdaea in term 1.
I20260812 07:09:49.841391 32677 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3bf80683ae714a0cacf072bd357cdaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a470d4a74ac4e2c8f0b777f50f39500" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:09:49.852691 32710 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bf80683ae714a0cacf072bd357cdaea: Generated new TSK 0
I20260812 07:09:49.853808 32710 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:09:49.906548 32728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a470d4a74ac4e2c8f0b777f50f39500 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bf80683ae714a0cacf072bd357cdaea. Latest consensus state: current_term: 1 leader_uuid: "3bf80683ae714a0cacf072bd357cdaea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bf80683ae714a0cacf072bd357cdaea" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 36257 } } peers { permanent_uuid: "8a470d4a74ac4e2c8f0b777f50f39500" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 34701 } } peers { permanent_uuid: "53071e25f9434e8ca88182548146bd7e" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 42729 } } }
I20260812 07:09:49.906919 32728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a470d4a74ac4e2c8f0b777f50f39500 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:09:49.907249 32735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a470d4a74ac4e2c8f0b777f50f39500 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3bf80683ae714a0cacf072bd357cdaea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bf80683ae714a0cacf072bd357cdaea" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 36257 } } peers { permanent_uuid: "8a470d4a74ac4e2c8f0b777f50f39500" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 34701 } } peers { permanent_uuid: "53071e25f9434e8ca88182548146bd7e" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 42729 } } }
I20260812 07:09:49.907521 32735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a470d4a74ac4e2c8f0b777f50f39500 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:09:49.908427 32705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bf80683ae714a0cacf072bd357cdaea [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3bf80683ae714a0cacf072bd357cdaea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bf80683ae714a0cacf072bd357cdaea" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 36257 } } peers { permanent_uuid: "8a470d4a74ac4e2c8f0b777f50f39500" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 34701 } } peers { permanent_uuid: "53071e25f9434e8ca88182548146bd7e" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 42729 } } }
I20260812 07:09:49.908811 32705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bf80683ae714a0cacf072bd357cdaea [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:49.953063 32749 catalog_manager.cc:1259] Loaded cluster ID: 14642d29f9a04bfeb68bf99ccbc003a3
I20260812 07:09:49.953189 32749 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8a470d4a74ac4e2c8f0b777f50f39500: loading cluster ID for follower catalog manager: success
I20260812 07:09:49.957142 32749 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8a470d4a74ac4e2c8f0b777f50f39500: acquiring CA information for follower catalog manager: success
I20260812 07:09:49.959843 32749 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8a470d4a74ac4e2c8f0b777f50f39500: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 07:09:49.980551 6388 mini_kdc.cc:348] Time spent kdestroy: real 0.115s user 0.000s sys 0.034s
W20260812 07:09:50.016898 32675 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:09:50.006094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53376 (local address 127.6.61.60:42729)
0812 07:09:50.006273 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:50.006280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:50.014491 (+ 8211us) server_negotiation.cc:407] Connection header received
0812 07:09:50.014904 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:50.014911 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:50.015030 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:50.015119 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:50.016595 (+ 1476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53376: BlockingRecv error: recv got EOF from 127.0.0.1:53376 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260812 07:09:50.020195 32673 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:09:50.006010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55796 (local address 127.6.61.62:36257)
0812 07:09:50.006259 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:50.006266 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:50.018209 (+ 11943us) server_negotiation.cc:407] Connection header received
0812 07:09:50.018497 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:50.018503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:50.018626 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:50.018706 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:50.019935 (+ 1229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55796: BlockingRecv error: recv got EOF from 127.0.0.1:55796 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260812 07:09:50.032184 32590 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:09:50.026758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50580 (local address 127.6.61.61:34701)
0812 07:09:50.026963 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:50.026974 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:50.026995 (+ 21us) server_negotiation.cc:407] Connection header received
0812 07:09:50.027238 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:50.027245 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:50.027371 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:50.027502 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:50.028760 (+ 1258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50580: BlockingRecv error: recv got EOF from 127.0.0.1:50580 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 07:09:50.046872 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:50.038448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50590 (local address 127.6.61.61:34701)
0812 07:09:50.038641 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:50.038648 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:50.039048 (+ 400us) server_negotiation.cc:407] Connection header received
0812 07:09:50.039308 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:50.039314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:50.039532 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:50.039708 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:09:50.040220 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:50.041402 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:50.043551 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:50.044291 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:50.044436 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:50.045076 (+ 640us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:09:50.045876 (+ 800us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:09:50.046024 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:50.046165 (+ 141us) server_negotiation.cc:299] Negotiation successful
0812 07:09:50.046296 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260812 07:09:50.069345 32675 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:09:50.062531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53390 (local address 127.6.61.60:42729)
0812 07:09:50.062943 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:50.062949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:50.067444 (+ 4495us) server_negotiation.cc:407] Connection header received
0812 07:09:50.067739 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:50.067746 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:50.067899 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:50.068031 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:50.069085 (+ 1054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53390: BlockingRecv error: recv got EOF from 127.0.0.1:53390 (error 108)
Metrics: {"server-negotiator.queue_time_us":198}
I20260812 07:09:50.083732 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:50.075377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50594 (local address 127.6.61.61:34701)
0812 07:09:50.075629 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:50.075635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:50.076266 (+ 631us) server_negotiation.cc:407] Connection header received
0812 07:09:50.076489 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:50.076496 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:50.076591 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:50.076698 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:09:50.077251 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:50.078593 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:50.081125 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:50.082032 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:50.082217 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:50.082732 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:09:50.082948 (+ 216us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:09:50.083062 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:50.083254 (+ 192us) server_negotiation.cc:299] Negotiation successful
0812 07:09:50.083356 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260812 07:09:50.111390 32675 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:09:50.108715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53396 (local address 127.6.61.60:42729)
0812 07:09:50.108916 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:50.108923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:50.109571 (+ 648us) server_negotiation.cc:407] Connection header received
0812 07:09:50.109784 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:50.109791 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:50.109969 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:50.110100 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:50.111096 (+ 996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53396: BlockingRecv error: recv got EOF from 127.0.0.1:53396 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 07:09:50.125332 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:50.117016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50606 (local address 127.6.61.61:34701)
0812 07:09:50.117217 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:50.117224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:50.117873 (+ 649us) server_negotiation.cc:407] Connection header received
0812 07:09:50.118278 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:50.118287 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:50.118455 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:50.118591 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:09:50.119597 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:50.120854 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:50.122951 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:50.123668 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:50.123830 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:50.124357 (+ 527us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:09:50.124550 (+ 193us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:09:50.124660 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:50.124845 (+ 185us) server_negotiation.cc:299] Negotiation successful
0812 07:09:50.124941 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260812 07:09:50.142393 32675 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:09:50.140354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53404 (local address 127.6.61.60:42729)
0812 07:09:50.140551 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:50.140558 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:50.140606 (+ 48us) server_negotiation.cc:407] Connection header received
0812 07:09:50.140922 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:50.140929 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:50.141043 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:50.141164 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:50.142109 (+ 945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53404: BlockingRecv error: recv got EOF from 127.0.0.1:53404 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 07:09:50.157778 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:50.150198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50622 (local address 127.6.61.61:34701)
0812 07:09:50.150421 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:50.150428 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:50.150970 (+ 542us) server_negotiation.cc:407] Connection header received
0812 07:09:50.151179 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:50.151186 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:50.151293 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:50.151401 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:09:50.151934 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:50.153274 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:50.155456 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:50.156228 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:50.156381 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:50.156934 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:09:50.157081 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:09:50.157192 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:50.157374 (+ 182us) server_negotiation.cc:299] Negotiation successful
0812 07:09:50.157469 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260812 07:09:50.172302 32675 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:09:50.168340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53412 (local address 127.6.61.60:42729)
0812 07:09:50.168653 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:50.168661 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:50.170492 (+ 1831us) server_negotiation.cc:407] Connection header received
0812 07:09:50.170839 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:50.170846 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:50.170953 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:50.171072 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:50.172065 (+ 993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53412: BlockingRecv error: recv got EOF from 127.0.0.1:53412 (error 108)
Metrics: {"server-negotiator.queue_time_us":173}
I20260812 07:09:50.192269 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:50.184499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50626 (local address 127.6.61.61:34701)
0812 07:09:50.184722 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:50.184730 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:50.185336 (+ 606us) server_negotiation.cc:407] Connection header received
0812 07:09:50.185572 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:50.185578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:50.185673 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:50.185793 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:09:50.186373 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:50.187547 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:50.189645 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:50.190459 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:50.190595 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:50.191271 (+ 676us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:09:50.191453 (+ 182us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:09:50.191553 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:50.191765 (+ 212us) server_negotiation.cc:299] Negotiation successful
0812 07:09:50.191857 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260812 07:09:50.210597 32675 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:09:50.204081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53420 (local address 127.6.61.60:42729)
0812 07:09:50.204299 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:50.204306 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:50.204387 (+ 81us) server_negotiation.cc:407] Connection header received
0812 07:09:50.204707 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:50.204713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:50.204846 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:50.204979 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:50.210284 (+ 5305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53420: BlockingRecv error: recv got EOF from 127.0.0.1:53420 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 07:09:50.234908 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:50.227150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50636 (local address 127.6.61.61:34701)
0812 07:09:50.227380 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:50.227387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:50.227966 (+ 579us) server_negotiation.cc:407] Connection header received
0812 07:09:50.228258 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:50.228264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:50.228362 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:50.228473 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:09:50.229110 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:50.230465 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:50.232582 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:50.233340 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:50.233490 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:50.234042 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:09:50.234188 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:09:50.234296 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:50.234494 (+ 198us) server_negotiation.cc:299] Negotiation successful
0812 07:09:50.234598 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260812 07:09:50.244613 32675 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:09:50.242237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53422 (local address 127.6.61.60:42729)
0812 07:09:50.242425 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:50.242432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:50.242977 (+ 545us) server_negotiation.cc:407] Connection header received
0812 07:09:50.243163 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:50.243168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:50.243268 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:50.243393 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:50.244372 (+ 979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53422: BlockingRecv error: recv got EOF from 127.0.0.1:53422 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 07:09:50.290457 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:50.283729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50652 (local address 127.6.61.61:34701)
0812 07:09:50.283967 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:50.283974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:50.284358 (+ 384us) server_negotiation.cc:407] Connection header received
0812 07:09:50.284587 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:50.284591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:50.284679 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:50.284787 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:09:50.285331 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:50.286221 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:50.288294 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:50.288818 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:50.288920 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:50.289479 (+ 559us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:09:50.289589 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:09:50.289765 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:50.289932 (+ 167us) server_negotiation.cc:299] Negotiation successful
0812 07:09:50.290069 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260812 07:09:50.299733 32675 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:09:50.297304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53424 (local address 127.6.61.60:42729)
0812 07:09:50.297555 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:50.297562 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:50.297908 (+ 346us) server_negotiation.cc:407] Connection header received
0812 07:09:50.298194 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:50.298200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:50.298284 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:50.298375 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:50.299446 (+ 1071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53424: BlockingRecv error: recv got EOF from 127.0.0.1:53424 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 07:09:50.375562 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:50.368144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50668 (local address 127.6.61.61:34701)
0812 07:09:50.368361 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:50.368369 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:50.369007 (+ 638us) server_negotiation.cc:407] Connection header received
0812 07:09:50.369331 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:50.369336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:50.369410 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:50.369504 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:09:50.370187 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:50.371037 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:50.373333 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:50.373888 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:50.374008 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:50.374573 (+ 565us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:09:50.374720 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:09:50.374795 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:50.375054 (+ 259us) server_negotiation.cc:299] Negotiation successful
0812 07:09:50.375151 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260812 07:09:50.384850 32675 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:09:50.382717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53430 (local address 127.6.61.60:42729)
0812 07:09:50.382960 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:50.382969 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:50.383299 (+ 330us) server_negotiation.cc:407] Connection header received
0812 07:09:50.383533 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:50.383538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:50.383621 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:50.383735 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:50.384571 (+ 836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53430: BlockingRecv error: recv got EOF from 127.0.0.1:53430 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 07:09:50.522604 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:50.516644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50676 (local address 127.6.61.61:34701)
0812 07:09:50.516841 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:50.516846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:50.517224 (+ 378us) server_negotiation.cc:407] Connection header received
0812 07:09:50.517454 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:50.517460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:50.517554 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:50.517687 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:09:50.518359 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:50.519081 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:50.520570 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:50.521103 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:50.521212 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:50.521720 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:09:50.521939 (+ 219us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:09:50.522018 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:50.522264 (+ 246us) server_negotiation.cc:299] Negotiation successful
0812 07:09:50.522342 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260812 07:09:50.531388 32675 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:09:50.528973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53434 (local address 127.6.61.60:42729)
0812 07:09:50.529238 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:50.529245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:50.529517 (+ 272us) server_negotiation.cc:407] Connection header received
0812 07:09:50.529675 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:50.529679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:50.529757 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:50.529938 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:50.531086 (+ 1148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53434: BlockingRecv error: recv got EOF from 127.0.0.1:53434 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
I20260812 07:09:50.796393 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:50.791171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50678 (local address 127.6.61.61:34701)
0812 07:09:50.791403 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:50.791408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:50.791714 (+ 306us) server_negotiation.cc:407] Connection header received
0812 07:09:50.791896 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:50.791900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:50.791991 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:50.792094 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:09:50.792509 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:50.793237 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:50.794772 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:50.795173 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:50.795304 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:50.795737 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:09:50.795857 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:09:50.795929 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:50.796084 (+ 155us) server_negotiation.cc:299] Negotiation successful
0812 07:09:50.796155 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260812 07:09:50.805855 32675 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:09:50.803746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53444 (local address 127.6.61.60:42729)
0812 07:09:50.804020 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:50.804028 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:50.804200 (+ 172us) server_negotiation.cc:407] Connection header received
0812 07:09:50.804452 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:50.804458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:50.804548 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:50.804665 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:50.805571 (+ 906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53444: BlockingRecv error: recv got EOF from 127.0.0.1:53444 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20260812 07:09:50.812429 32727 catalog_manager.cc:1259] Loaded cluster ID: 14642d29f9a04bfeb68bf99ccbc003a3
I20260812 07:09:50.812601 32727 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 53071e25f9434e8ca88182548146bd7e: loading cluster ID for follower catalog manager: success
I20260812 07:09:50.815536 32727 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 53071e25f9434e8ca88182548146bd7e: acquiring CA information for follower catalog manager: success
I20260812 07:09:50.817504 32727 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 53071e25f9434e8ca88182548146bd7e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 07:09:51.328780 412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:51.322923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50694 (local address 127.6.61.61:34701)
0812 07:09:51.323529 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:51.323536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:51.323841 (+ 305us) server_negotiation.cc:407] Connection header received
0812 07:09:51.323993 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:51.323996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:51.324066 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:51.324154 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:09:51.324544 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:51.325302 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:51.326863 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:51.327363 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:51.327467 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:51.327988 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:09:51.328099 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:09:51.328210 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:51.328358 (+ 148us) server_negotiation.cc:299] Negotiation successful
0812 07:09:51.328428 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":326,"threads_started":1}
I20260812 07:09:51.342973 419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:51.336309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53452 (local address 127.6.61.60:42729)
0812 07:09:51.336863 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:51.336868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:51.336887 (+ 19us) server_negotiation.cc:407] Connection header received
0812 07:09:51.337112 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:51.337117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:51.337294 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:51.337421 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:09:51.338009 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:51.338821 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:51.340334 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:51.340756 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:51.340866 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:51.341487 (+ 621us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:09:51.342203 (+ 716us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:09:51.342323 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:51.342512 (+ 189us) server_negotiation.cc:299] Negotiation successful
0812 07:09:51.342594 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":306,"threads_started":1}
I20260812 07:09:51.359530 412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:51.352009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50702 (local address 127.6.61.61:34701)
0812 07:09:51.352241 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:51.352250 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:51.353364 (+ 1114us) server_negotiation.cc:407] Connection header received
0812 07:09:51.353576 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:51.353582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:51.353659 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:51.353771 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:09:51.354417 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:51.355589 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:51.357317 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:51.357925 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:51.358051 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:51.358694 (+ 643us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:09:51.358851 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:09:51.358966 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:51.359112 (+ 146us) server_negotiation.cc:299] Negotiation successful
0812 07:09:51.359197 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 07:09:51.359886 419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:51.350789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53456 (local address 127.6.61.60:42729)
0812 07:09:51.352539 (+ 1750us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:51.352549 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:51.352577 (+ 28us) server_negotiation.cc:407] Connection header received
0812 07:09:51.352825 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:51.352834 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:51.352956 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:51.353077 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:09:51.353902 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:51.355008 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:51.357605 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:51.358270 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:51.358410 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:51.358959 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:09:51.359119 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:09:51.359222 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:51.359466 (+ 244us) server_negotiation.cc:299] Negotiation successful
0812 07:09:51.359565 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1515}
I20260812 07:09:51.361387 425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:51.350645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55808 (local address 127.6.61.62:36257)
0812 07:09:51.354048 (+ 3403us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:51.354055 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:51.354076 (+ 21us) server_negotiation.cc:407] Connection header received
0812 07:09:51.354208 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:51.354214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:51.354449 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:51.354633 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:09:51.355145 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:51.355964 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:51.358474 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:51.359111 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:51.359847 (+ 736us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:51.360103 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:09:51.360660 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:09:51.360844 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:51.360976 (+ 132us) server_negotiation.cc:299] Negotiation successful
0812 07:09:51.361074 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2781,"thread_start_us":2584,"threads_started":1}
I20260812 07:09:51.367293 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 32421
I20260812 07:09:51.368507 32491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:09:51.492074 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 32421
I20260812 07:09:51.526392 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 32505
I20260812 07:09:51.527667 32580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:09:51.644507 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 32505
I20260812 07:09:51.672976 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 32593
I20260812 07:09:51.674431 32664 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:09:51.783862 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 32593
2026-08-12T07:09:51Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5476 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:09:51 dist-test-slave-69rg krb5kdc[435](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:09:51 dist-test-slave-69rg krb5kdc[435](info): set up 2 sockets
Aug 12 07:09:51 dist-test-slave-69rg krb5kdc[435](info): commencing operation
krb5kdc: starting...
W20260812 07:09:53.049144 6388 mini_kdc.cc:121] Time spent starting KDC: real 1.219s user 0.000s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/krb5kdc/test-user.keytab.
Aug 12 07:09:53 dist-test-slave-69rg krb5kdc[435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:09:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:09:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
Entry for principal kudu/127.6.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab.
I20260812 07:09:53.410247 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:35063
--webserver_interface=127.6.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:45573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:35063,127.6.61.61:40703,127.6.61.60:36781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:09:53.762940 450 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 07:09:53.763334 450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:09:53.763445 450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:09:53.763602 450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:09:53.771289 450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:09:53.771404 450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:09:53.771493 450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:09:53.771595 450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:09:53.771726 450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:09:53.782148 450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:45573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:35063,127.6.61.61:40703,127.6.61.60:36781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.62.keytab
--principal=kudu/127.6.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.62:35063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:09:53.783552 450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:09:53.784907 450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:09:53.797833 457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:53.797861 456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:53.797758 459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:09:53.798768 450 server_base.cc:1034] running on GCE node
I20260812 07:09:53.799707 450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:09:53.801231 450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:09:53.802523 450 hybrid_clock.cc:648] HybridClock initialized: now 1786518593802445 us; error 69 us; skew 500 ppm
I20260812 07:09:53.802925 450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:09:53 dist-test-slave-69rg krb5kdc[435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518593, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:09:53.810483 450 init.cc:377] Logged in from keytab as kudu/127.6.61.62@KRBTEST.COM (short username kudu)
I20260812 07:09:53.812163 450 webserver.cc:460] Webserver started at http://127.6.61.62:38559/ using document root <none> and password file <none>
I20260812 07:09:53.812876 450 fs_manager.cc:359] Metadata directory not provided
I20260812 07:09:53.812971 450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:09:53.813247 450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:09:53.816342 450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-0/data/instance:
uuid: "9889f2c00e9744098278434e2f800153"
format_stamp: "Formatted at 2026-08-12 07:09:53 on dist-test-slave-69rg"
server_key: "abbdaa93943949e8a652a6d5bfb5de62"
server_key_iv: "09751ba22583af8ee0cc3163cb6ca5de"
server_key_version: "encryptionkey@0"
I20260812 07:09:53.817111 450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-0/wal/instance:
uuid: "9889f2c00e9744098278434e2f800153"
format_stamp: "Formatted at 2026-08-12 07:09:53 on dist-test-slave-69rg"
server_key: "abbdaa93943949e8a652a6d5bfb5de62"
server_key_iv: "09751ba22583af8ee0cc3163cb6ca5de"
server_key_version: "encryptionkey@0"
I20260812 07:09:53.822486 450 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 07:09:53.826222 468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:09:53.827673 450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:09:53.827870 450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-0/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-0/wal
uuid: "9889f2c00e9744098278434e2f800153"
format_stamp: "Formatted at 2026-08-12 07:09:53 on dist-test-slave-69rg"
server_key: "abbdaa93943949e8a652a6d5bfb5de62"
server_key_iv: "09751ba22583af8ee0cc3163cb6ca5de"
server_key_version: "encryptionkey@0"
I20260812 07:09:53.828074 450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:09:53.842300 450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:09:53.846863 450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:09:53.847113 450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:09:53.867615 450 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.62:35063
I20260812 07:09:53.867655 523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.62:35063 every 8 connection(s)
I20260812 07:09:53.869059 450 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-0/data/info.pb
I20260812 07:09:53.872987 524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:09:53.877331 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 450
I20260812 07:09:53.877542 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-0/wal/instance
I20260812 07:09:53.878175 6388 external_mini_cluster.cc:1428] Setting key 819780b9be1363c28c788cff959ff448
I20260812 07:09:53.881389 524 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35063 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 07:09:53 dist-test-slave-69rg krb5kdc[435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518593, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:09:53.909021 529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:53.886909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42788 (local address 127.6.61.62:35063)
0812 07:09:53.887734 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:53.887745 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:53.887804 (+ 59us) server_negotiation.cc:407] Connection header received
0812 07:09:53.889097 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:53.889107 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:53.889432 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:53.889664 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:53.890924 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:53.892872 (+ 1948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:53.895182 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:53.895786 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:53.895932 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:53.900939 (+ 5007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:53.900949 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:53.900965 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:53.901003 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:53.904141 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:53.905250 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:53.905258 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:53.905260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:53.905435 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:53.906027 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:53.906031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:53.906034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:53.906598 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:53.906771 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:53.907566 (+ 795us) server_negotiation.cc:299] Negotiation successful
0812 07:09:53.907898 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":563,"thread_start_us":401,"threads_started":1}
I20260812 07:09:53.909092 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:53.885512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:35063 (local address 127.0.0.1:42788)
0812 07:09:53.886448 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:53.886495 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 07:09:53.888693 (+ 2198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:09:53.889700 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:09:53.889706 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:09:53.890335 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:09:53.890688 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:53.890706 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:53.893061 (+ 2355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:53.893065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:53.895004 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:53.895024 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:53.896057 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:53.896068 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:53.896321 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:53.897126 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:09:53.897162 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:09:53.900564 (+ 3402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:09:53.904342 (+ 3778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:53.904360 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:53.904378 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:53.905049 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:53.905574 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:53.905581 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:53.905587 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:53.905884 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:53.906761 (+ 877us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:09:53.906773 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:09:53.907051 (+ 278us) client_negotiation.cc:769] Sending connection context
0812 07:09:53.907384 (+ 333us) client_negotiation.cc:240] Negotiation successful
0812 07:09:53.908079 (+ 695us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":84,"thread_start_us":438,"threads_started":1}
I20260812 07:09:53.912743 524 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.61.61" port: 40703 } has no permanent_uuid. Determining permanent_uuid...
I20260812 07:09:53.914170 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:53.913546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:40703 (local address 127.0.0.1:37182)
0812 07:09:53.913857 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:53.913999 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.61:40703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260812 07:09:53.914567 469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.61.61:40703: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.6.61.61@KRBTEST.COM; defaulting to no policy
I20260812 07:09:53.916249 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:53.915855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:40703 (local address 127.0.0.1:37184)
0812 07:09:53.915990 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:53.916069 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.61:40703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260812 07:09:53.916479 524 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.61:40703: Network error: Client connection negotiation failed: client connection to 127.6.61.61:40703: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.61.keytab.
Entry for principal kudu/127.6.61.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.61.keytab.
I20260812 07:09:53.957269 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.61:40703
--webserver_interface=127.6.61.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.61.keytab
--principal=kudu/127.6.61.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:45573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:35063,127.6.61.61:40703,127.6.61.60:36781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:09:53.965903 524 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.61" port: 40703 } attempt: 1
I20260812 07:09:53.967358 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:53.966751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:40703 (local address 127.0.0.1:37186)
0812 07:09:53.967010 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:53.967145 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.61:40703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260812 07:09:53.968564 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:53.968101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:40703 (local address 127.0.0.1:37196)
0812 07:09:53.968288 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:53.968385 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.61:40703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260812 07:09:53.968822 524 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.61:40703: Network error: Client connection negotiation failed: client connection to 127.6.61.61:40703: connect: Connection refused (error 111)
I20260812 07:09:54.037129 524 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.61" port: 40703 } attempt: 2
I20260812 07:09:54.038553 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:54.037980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:40703 (local address 127.0.0.1:37210)
0812 07:09:54.038304 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:54.038401 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.61:40703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260812 07:09:54.039774 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:54.039331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:40703 (local address 127.0.0.1:37220)
0812 07:09:54.039518 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:54.039635 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.61:40703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260812 07:09:54.039985 524 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.61:40703: Network error: Client connection negotiation failed: client connection to 127.6.61.61:40703: connect: Connection refused (error 111)
I20260812 07:09:54.131281 524 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.61" port: 40703 } attempt: 3
I20260812 07:09:54.132601 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:54.132089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:40703 (local address 127.0.0.1:37234)
0812 07:09:54.132337 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:54.132455 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.61:40703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260812 07:09:54.134007 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:54.133548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:40703 (local address 127.0.0.1:37244)
0812 07:09:54.133722 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:54.133793 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.61:40703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260812 07:09:54.134275 524 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.61:40703: Network error: Client connection negotiation failed: client connection to 127.6.61.61:40703: connect: Connection refused (error 111)
I20260812 07:09:54.277585 524 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.61" port: 40703 } attempt: 4
I20260812 07:09:54.279064 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:54.278486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:40703 (local address 127.0.0.1:37246)
0812 07:09:54.278846 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:54.278933 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.61:40703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196}
I20260812 07:09:54.280287 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:54.279807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:40703 (local address 127.0.0.1:37248)
0812 07:09:54.279993 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:54.280132 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.61:40703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260812 07:09:54.280551 524 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.61:40703: Network error: Client connection negotiation failed: client connection to 127.6.61.61:40703: connect: Connection refused (error 111)
W20260812 07:09:54.313651 533 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 07:09:54.314049 533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:09:54.314128 533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:09:54.314213 533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:09:54.321353 533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:09:54.321440 533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:09:54.321498 533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:09:54.321542 533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:09:54.321604 533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:09:54.331355 533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:45573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:35063,127.6.61.61:40703,127.6.61.60:36781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.61.keytab
--principal=kudu/127.6.61.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.61:40703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.61.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:09:54.332646 533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:09:54.334070 533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:09:54.346552 539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:54.346709 541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:54.347139 538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:09:54.347589 533 server_base.cc:1034] running on GCE node
I20260812 07:09:54.348475 533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:09:54.350076 533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:09:54.351326 533 hybrid_clock.cc:648] HybridClock initialized: now 1786518594351247 us; error 60 us; skew 500 ppm
I20260812 07:09:54.351663 533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:09:54 dist-test-slave-69rg krb5kdc[435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518594, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:09:54.355057 533 init.cc:377] Logged in from keytab as kudu/127.6.61.61@KRBTEST.COM (short username kudu)
I20260812 07:09:54.356698 533 webserver.cc:460] Webserver started at http://127.6.61.61:34261/ using document root <none> and password file <none>
I20260812 07:09:54.357378 533 fs_manager.cc:359] Metadata directory not provided
I20260812 07:09:54.357468 533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:09:54.357733 533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:09:54.360752 533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-1/data/instance:
uuid: "006e588b5094425eac49d60068a86573"
format_stamp: "Formatted at 2026-08-12 07:09:54 on dist-test-slave-69rg"
server_key: "a20ce6b819061a03e4476732919e5dc5"
server_key_iv: "a944aea6055279f857469595072a1bd2"
server_key_version: "encryptionkey@0"
I20260812 07:09:54.361510 533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-1/wal/instance:
uuid: "006e588b5094425eac49d60068a86573"
format_stamp: "Formatted at 2026-08-12 07:09:54 on dist-test-slave-69rg"
server_key: "a20ce6b819061a03e4476732919e5dc5"
server_key_iv: "a944aea6055279f857469595072a1bd2"
server_key_version: "encryptionkey@0"
I20260812 07:09:54.366799 533 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 07:09:54.370522 550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:09:54.372269 533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:09:54.372501 533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-1/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-1/wal
uuid: "006e588b5094425eac49d60068a86573"
format_stamp: "Formatted at 2026-08-12 07:09:54 on dist-test-slave-69rg"
server_key: "a20ce6b819061a03e4476732919e5dc5"
server_key_iv: "a944aea6055279f857469595072a1bd2"
server_key_version: "encryptionkey@0"
I20260812 07:09:54.372735 533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:09:54.397120 533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:09:54.402329 533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:09:54.402662 533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:09:54.422763 533 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.61:40703
I20260812 07:09:54.422787 605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.61:40703 every 8 connection(s)
I20260812 07:09:54.424212 533 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-1/data/info.pb
I20260812 07:09:54.428186 606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:09:54.433530 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 533
I20260812 07:09:54.433736 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-1/wal/instance
I20260812 07:09:54.434226 6388 external_mini_cluster.cc:1428] Setting key 8826cc92332c3029ce6d4d18bbb477ef
I20260812 07:09:54.435629 606 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35063 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 07:09:54 dist-test-slave-69rg krb5kdc[435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518594, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.61@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:09:54.460825 609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:54.438441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42790 (local address 127.6.61.62:35063)
0812 07:09:54.438986 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:54.438995 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:54.440044 (+ 1049us) server_negotiation.cc:407] Connection header received
0812 07:09:54.442992 (+ 2948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:54.442997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:54.443076 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:54.443196 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:54.445358 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:54.446260 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:54.448714 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:54.449142 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:54.449255 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:54.454201 (+ 4946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:54.454208 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:54.454211 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:54.454261 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:54.457173 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:54.458272 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:54.458276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:54.458279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:54.458407 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:54.458925 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:54.458928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:54.458930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:54.459261 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:54.459378 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:54.460359 (+ 981us) server_negotiation.cc:299] Negotiation successful
0812 07:09:54.460540 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":258,"threads_started":1}
I20260812 07:09:54.462509 610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:54.438997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:35063 (local address 127.0.0.1:42790)
0812 07:09:54.439830 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:54.439889 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 07:09:54.442657 (+ 2768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:09:54.443451 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:09:54.443463 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:09:54.444296 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:09:54.445118 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:54.445146 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:54.446432 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:54.446440 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:54.448517 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:54.448535 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:54.449417 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:54.449426 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:54.449767 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:54.450603 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:09:54.450641 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:09:54.453916 (+ 3275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:09:54.457359 (+ 3443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:54.457373 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:54.457395 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:54.458113 (+ 718us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:54.458523 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:54.458531 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:54.458537 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:54.458799 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:54.459412 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:09:54.459423 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:09:54.459922 (+ 499us) client_negotiation.cc:769] Sending connection context
0812 07:09:54.460257 (+ 335us) client_negotiation.cc:240] Negotiation successful
0812 07:09:54.460900 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":333,"threads_started":1}
I20260812 07:09:54.464417 606 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.61.61" port: 40703 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.61.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.61.60@KRBTEST.COM" created.
Aug 12 07:09:54 dist-test-slave-69rg krb5kdc[435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518594, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.61@KRBTEST.COM for kudu/127.6.61.61@KRBTEST.COM
I20260812 07:09:54.482604 610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:54.465255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:40703 (local address 127.0.0.1:37258)
0812 07:09:54.465463 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:54.465482 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 07:09:54.465582 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:09:54.466905 (+ 1323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:09:54.466910 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:09:54.467462 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:09:54.467612 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:54.467631 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:54.469230 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:54.469239 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:54.470301 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:54.470314 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:54.471299 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:54.471305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:54.471505 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:54.472156 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:09:54.472181 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:09:54.474868 (+ 2687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:09:54.479897 (+ 5029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:54.479902 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:54.479905 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:54.480288 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:54.480780 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:54.480784 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:54.480787 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:54.480918 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:54.481739 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:09:54.481743 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:09:54.481976 (+ 233us) client_negotiation.cc:769] Sending connection context
0812 07:09:54.482097 (+ 121us) client_negotiation.cc:240] Negotiation successful
0812 07:09:54.482322 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260812 07:09:54.483091 613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:54.465534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37258 (local address 127.6.61.61:40703)
0812 07:09:54.466412 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:54.466422 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:54.466463 (+ 41us) server_negotiation.cc:407] Connection header received
0812 07:09:54.466516 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:54.466522 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:54.466719 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:54.466924 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:54.467834 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:54.469057 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:54.470452 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:54.471098 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:54.471275 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:54.475160 (+ 3885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:54.475185 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:54.475213 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:54.475273 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:54.479711 (+ 4438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:54.480445 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:54.480457 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:54.480463 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:54.480621 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:54.481035 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:54.481043 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:54.481049 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:54.481577 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:54.481808 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:54.482293 (+ 485us) server_negotiation.cc:299] Negotiation successful
0812 07:09:54.482591 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":475,"threads_started":1}
I20260812 07:09:54.485946 606 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.61.60" port: 36781 } has no permanent_uuid. Determining permanent_uuid...
I20260812 07:09:54.487232 610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:54.486761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:36781 (local address 127.0.0.1:49196)
0812 07:09:54.486972 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:54.487087 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:36781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260812 07:09:54.487550 554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.61.60:36781: connect: Connection refused (error 111)
I20260812 07:09:54.489199 610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:54.488877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:36781 (local address 127.0.0.1:49212)
0812 07:09:54.489019 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:54.489100 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:36781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260812 07:09:54.489418 606 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.60:36781: Network error: Client connection negotiation failed: client connection to 127.6.61.60:36781: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.61.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.60.keytab.
Entry for principal kudu/127.6.61.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.60.keytab.
I20260812 07:09:54.515527 6388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6k_OM2/build/asan/bin/kudu
/tmp/dist-test-task6k_OM2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.60:36781
--webserver_interface=127.6.61.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.60.keytab
--principal=kudu/127.6.61.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.61.20:45573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.61.62:35063,127.6.61.61:40703,127.6.61.60:36781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:09:54.538758 606 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.60" port: 36781 } attempt: 1
I20260812 07:09:54.540114 610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:54.539555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:36781 (local address 127.0.0.1:49216)
0812 07:09:54.539824 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:54.539922 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:36781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260812 07:09:54.541435 610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:54.540909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:36781 (local address 127.0.0.1:49218)
0812 07:09:54.541126 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:54.541266 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:36781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260812 07:09:54.541739 606 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.60:36781: Network error: Client connection negotiation failed: client connection to 127.6.61.60:36781: connect: Connection refused (error 111)
I20260812 07:09:54.579851 524 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.61" port: 40703 } attempt: 5
Aug 12 07:09:54 dist-test-slave-69rg krb5kdc[435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518593, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for kudu/127.6.61.61@KRBTEST.COM
I20260812 07:09:54.594290 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:54.580738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:40703 (local address 127.0.0.1:37266)
0812 07:09:54.580979 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:54.581004 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 07:09:54.581149 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:09:54.581576 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:09:54.581596 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:09:54.582088 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:09:54.582283 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:54.582292 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:54.583324 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:54.583334 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:54.584280 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:54.584293 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:54.585047 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:54.585053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:54.585260 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:54.586017 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:09:54.586037 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:09:54.588351 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:09:54.591625 (+ 3274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:54.591630 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:54.591634 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:54.592167 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:54.592572 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:54.592577 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:54.592581 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:54.592775 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:54.593360 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:09:54.593365 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:09:54.593529 (+ 164us) client_negotiation.cc:769] Sending connection context
0812 07:09:54.593714 (+ 185us) client_negotiation.cc:240] Negotiation successful
0812 07:09:54.594039 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260812 07:09:54.594448 613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:54.580894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37266 (local address 127.6.61.61:40703)
0812 07:09:54.581081 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:54.581122 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:54.581145 (+ 23us) server_negotiation.cc:407] Connection header received
0812 07:09:54.581321 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:54.581329 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:54.581420 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:54.581533 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:54.582448 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:54.583154 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:54.584443 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:54.584894 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:54.585033 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:54.588563 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:54.588570 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:54.588573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:54.588607 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:54.591482 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:54.592295 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:54.592300 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:54.592302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:54.592436 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:54.592897 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:54.592903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:54.592906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:54.593232 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:54.593395 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:54.593870 (+ 475us) server_negotiation.cc:299] Negotiation successful
0812 07:09:54.594143 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 07:09:54.595567 524 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.61.60" port: 36781 } has no permanent_uuid. Determining permanent_uuid...
I20260812 07:09:54.596732 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:54.596342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:36781 (local address 127.0.0.1:49224)
0812 07:09:54.596505 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:54.596579 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:36781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260812 07:09:54.597802 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:54.597476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:36781 (local address 127.0.0.1:49226)
0812 07:09:54.597622 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:54.597711 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:36781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 07:09:54.598049 524 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.60:36781: Network error: Client connection negotiation failed: client connection to 127.6.61.60:36781: connect: Connection refused (error 111)
I20260812 07:09:54.609993 606 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.60" port: 36781 } attempt: 2
I20260812 07:09:54.611186 610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:54.610656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:36781 (local address 127.0.0.1:49234)
0812 07:09:54.610869 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:54.610997 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:36781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260812 07:09:54.612224 610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:54.611835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:36781 (local address 127.0.0.1:49242)
0812 07:09:54.611997 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:54.612084 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:36781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260812 07:09:54.612463 606 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.60:36781: Network error: Client connection negotiation failed: client connection to 127.6.61.60:36781: connect: Connection refused (error 111)
I20260812 07:09:54.649317 524 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.60" port: 36781 } attempt: 1
I20260812 07:09:54.650599 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:54.650088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:36781 (local address 127.0.0.1:49258)
0812 07:09:54.650310 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:54.650419 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:36781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260812 07:09:54.651715 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:54.651364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:36781 (local address 127.0.0.1:49266)
0812 07:09:54.651521 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:54.651600 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:36781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260812 07:09:54.651950 524 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.60:36781: Network error: Client connection negotiation failed: client connection to 127.6.61.60:36781: connect: Connection refused (error 111)
I20260812 07:09:54.703768 606 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.60" port: 36781 } attempt: 3
I20260812 07:09:54.705264 610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:54.704588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:36781 (local address 127.0.0.1:49274)
0812 07:09:54.704897 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:54.705027 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:36781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260812 07:09:54.706593 610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:54.706010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:36781 (local address 127.0.0.1:49280)
0812 07:09:54.706306 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:54.706419 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:36781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
W20260812 07:09:54.706890 606 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.60:36781: Network error: Client connection negotiation failed: client connection to 127.6.61.60:36781: connect: Connection refused (error 111)
I20260812 07:09:54.720167 524 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.60" port: 36781 } attempt: 2
I20260812 07:09:54.721429 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:54.720891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:36781 (local address 127.0.0.1:49286)
0812 07:09:54.721123 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:54.721237 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:36781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260812 07:09:54.722729 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:54.722316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:36781 (local address 127.0.0.1:49296)
0812 07:09:54.722504 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:54.722588 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:36781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260812 07:09:54.722960 524 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.60:36781: Network error: Client connection negotiation failed: client connection to 127.6.61.60:36781: connect: Connection refused (error 111)
I20260812 07:09:54.829198 524 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.60" port: 36781 } attempt: 3
I20260812 07:09:54.830497 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:54.829998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:36781 (local address 127.0.0.1:49302)
0812 07:09:54.830244 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:54.830338 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:36781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260812 07:09:54.831595 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:54.831178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:36781 (local address 127.0.0.1:49304)
0812 07:09:54.831430 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:54.831500 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:36781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260812 07:09:54.831831 524 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.60:36781: Network error: Client connection negotiation failed: client connection to 127.6.61.60:36781: connect: Connection refused (error 111)
I20260812 07:09:54.850155 606 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.60" port: 36781 } attempt: 4
I20260812 07:09:54.851495 610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:54.850860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:36781 (local address 127.0.0.1:49318)
0812 07:09:54.851154 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:54.851277 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:36781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260812 07:09:54.852603 610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:54.852172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:36781 (local address 127.0.0.1:49322)
0812 07:09:54.852355 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:54.852452 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.61.60:36781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260812 07:09:54.852901 606 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.61.60:36781: Network error: Client connection negotiation failed: client connection to 127.6.61.60:36781: connect: Connection refused (error 111)
W20260812 07:09:54.873081 616 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 07:09:54.873428 616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:09:54.873566 616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:09:54.873697 616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:09:54.881644 616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:09:54.881750 616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:09:54.881887 616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:09:54.881950 616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:09:54.882047 616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:09:54.892460 616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.61.20:45573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.61.62:35063,127.6.61.61:40703,127.6.61.60:36781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/krb5kdc/kudu_127.6.61.60.keytab
--principal=kudu/127.6.61.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.61.60:36781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.61.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 05:55:59 UTC on ccycloud.quasar-tbxajw.root.comops.site
build id 4581995
ASAN enabled
I20260812 07:09:54.893887 616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:09:54.895264 616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:09:54.907486 622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:54.907451 623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:09:54.908061 625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:09:54.908416 616 server_base.cc:1034] running on GCE node
I20260812 07:09:54.909211 616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:09:54.910796 616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:09:54.912050 616 hybrid_clock.cc:648] HybridClock initialized: now 1786518594912003 us; error 53 us; skew 500 ppm
I20260812 07:09:54.912428 616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 07:09:54 dist-test-slave-69rg krb5kdc[435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518594, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:09:54.915671 616 init.cc:377] Logged in from keytab as kudu/127.6.61.60@KRBTEST.COM (short username kudu)
I20260812 07:09:54.917161 616 webserver.cc:460] Webserver started at http://127.6.61.60:36985/ using document root <none> and password file <none>
I20260812 07:09:54.917943 616 fs_manager.cc:359] Metadata directory not provided
I20260812 07:09:54.918089 616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:09:54.918442 616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:09:54.921433 616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-2/data/instance:
uuid: "c0cd65fa649b462ca72e4f51dfcdf71c"
format_stamp: "Formatted at 2026-08-12 07:09:54 on dist-test-slave-69rg"
server_key: "74998fd7e5bdd79262bc7f07c8e23c17"
server_key_iv: "756390b109823fb8f919ecd75b2d946c"
server_key_version: "encryptionkey@0"
I20260812 07:09:54.922254 616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-2/wal/instance:
uuid: "c0cd65fa649b462ca72e4f51dfcdf71c"
format_stamp: "Formatted at 2026-08-12 07:09:54 on dist-test-slave-69rg"
server_key: "74998fd7e5bdd79262bc7f07c8e23c17"
server_key_iv: "756390b109823fb8f919ecd75b2d946c"
server_key_version: "encryptionkey@0"
I20260812 07:09:54.927270 616 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260812 07:09:54.930680 632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:09:54.931938 616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:09:54.932130 616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-2/data,/tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-2/wal
uuid: "c0cd65fa649b462ca72e4f51dfcdf71c"
format_stamp: "Formatted at 2026-08-12 07:09:54 on dist-test-slave-69rg"
server_key: "74998fd7e5bdd79262bc7f07c8e23c17"
server_key_iv: "756390b109823fb8f919ecd75b2d946c"
server_key_version: "encryptionkey@0"
I20260812 07:09:54.932355 616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:09:54.948601 616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:09:54.953194 616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:09:54.953500 616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:09:54.973582 616 rpc_server.cc:307] RPC server started. Bound to: 127.6.61.60:36781
I20260812 07:09:54.973603 688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.61.60:36781 every 8 connection(s)
I20260812 07:09:54.974979 616 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-2/data/info.pb
I20260812 07:09:54.978854 689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:09:54.981654 6388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6k_OM2/build/asan/bin/kudu as pid 616
I20260812 07:09:54.981937 6388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6k_OM2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518208686283-6388-0/minicluster-data/master-2/wal/instance
I20260812 07:09:54.982417 6388 external_mini_cluster.cc:1428] Setting key 5eb3a5fdcf97fdb84896552de2c8163d
I20260812 07:09:54.986645 689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35063 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 07:09:54 dist-test-slave-69rg krb5kdc[435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.60@KRBTEST.COM
Aug 12 07:09:54 dist-test-slave-69rg krb5kdc[435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518594, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.60@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:09:55.006920 695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:54.990250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42802 (local address 127.6.61.62:35063)
0812 07:09:54.990756 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:54.990762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:54.991355 (+ 593us) server_negotiation.cc:407] Connection header received
0812 07:09:54.991543 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:54.991548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:54.991658 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:54.991774 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:54.993228 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:54.994168 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:54.996469 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:54.996865 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:54.996976 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:55.000933 (+ 3957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:55.000942 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:55.000948 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:55.001004 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:55.003931 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:55.004696 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:55.004700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:55.004702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:55.004842 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:55.005245 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:55.005248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:55.005250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:55.005616 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:55.005781 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:55.006428 (+ 647us) server_negotiation.cc:299] Negotiation successful
0812 07:09:55.006633 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":254,"threads_started":1}
I20260812 07:09:55.007962 693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:54.984465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49334 (local address 127.6.61.60:36781)
0812 07:09:54.985241 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:54.985256 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:54.985307 (+ 51us) server_negotiation.cc:407] Connection header received
0812 07:09:54.987409 (+ 2102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:54.987440 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:54.988018 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:54.988457 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:54.990058 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:54.991844 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:54.993705 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:54.994605 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:54.994743 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:55.000130 (+ 5387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:55.000141 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:55.000159 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:55.000195 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:55.003180 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:55.003879 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:55.003894 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:55.003901 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:55.004101 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:55.004554 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:55.004560 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:55.004563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:55.005289 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:55.005499 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:55.006061 (+ 562us) server_negotiation.cc:299] Negotiation successful
0812 07:09:55.006534 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":547,"thread_start_us":320,"threads_started":1}
I20260812 07:09:55.007999 696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:54.990297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.62:35063 (local address 127.0.0.1:42802)
0812 07:09:54.991196 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:54.991250 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 07:09:54.991401 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:09:54.991865 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:09:54.991870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:09:54.992752 (+ 882us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:09:54.993059 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:54.993073 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:54.994371 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:54.994376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:54.996271 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:54.996292 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:54.997123 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:54.997134 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:54.997383 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:54.998181 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:09:54.998223 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:09:55.000739 (+ 2516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:09:55.004059 (+ 3320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:55.004069 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:55.004077 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:55.004554 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:55.004964 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:55.004968 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:55.004972 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:55.005120 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:55.005747 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:09:55.005754 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:09:55.006063 (+ 309us) client_negotiation.cc:769] Sending connection context
0812 07:09:55.006260 (+ 197us) client_negotiation.cc:240] Negotiation successful
0812 07:09:55.006664 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":156,"thread_start_us":485,"threads_started":1}
I20260812 07:09:55.009057 524 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.60" port: 36781 } attempt: 4
I20260812 07:09:55.009963 689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.61.61" port: 40703 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 07:09:55 dist-test-slave-69rg krb5kdc[435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518593, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.62@KRBTEST.COM for kudu/127.6.61.60@KRBTEST.COM
Aug 12 07:09:55 dist-test-slave-69rg krb5kdc[435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518594, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.60@KRBTEST.COM for kudu/127.6.61.61@KRBTEST.COM
Aug 12 07:09:55 dist-test-slave-69rg krb5kdc[435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.62@KRBTEST.COM
I20260812 07:09:55.026072 693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:55.009786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49336 (local address 127.6.61.60:36781)
0812 07:09:55.009984 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:55.009989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:55.010082 (+ 93us) server_negotiation.cc:407] Connection header received
0812 07:09:55.010252 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:55.010257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:55.010331 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:55.010503 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:55.011498 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.012470 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.013764 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.014496 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.014631 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:55.019662 (+ 5031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:55.019673 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:55.019677 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:55.019733 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:55.023028 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:55.023799 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:55.023808 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:55.023812 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:55.024027 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:55.024445 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:55.024452 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:55.024455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:55.024767 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:55.024923 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:55.025435 (+ 512us) server_negotiation.cc:299] Negotiation successful
0812 07:09:55.025684 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260812 07:09:55.026134 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:55.009698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:36781 (local address 127.0.0.1:49336)
0812 07:09:55.009983 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:55.010006 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 07:09:55.010126 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:09:55.010556 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:09:55.010564 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:09:55.011118 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:09:55.011330 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:55.011344 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.012619 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.012625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:55.013567 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:55.013579 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.014660 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.014667 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:55.014894 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:55.016481 (+ 1587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:09:55.016498 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:09:55.019439 (+ 2941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:09:55.023166 (+ 3727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:55.023171 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:55.023175 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:55.023638 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:55.024159 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:55.024170 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:55.024174 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:55.024323 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:55.024898 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:09:55.024903 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:09:55.025100 (+ 197us) client_negotiation.cc:769] Sending connection context
0812 07:09:55.025316 (+ 216us) client_negotiation.cc:240] Negotiation successful
0812 07:09:55.025675 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260812 07:09:55.026701 613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:55.011284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37272 (local address 127.6.61.61:40703)
0812 07:09:55.011476 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:55.011481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:55.011501 (+ 20us) server_negotiation.cc:407] Connection header received
0812 07:09:55.011563 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:55.011569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:55.011649 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:55.011762 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:55.012776 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.013846 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.015482 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.016126 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.016255 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:55.020090 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:55.020095 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:55.020098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:55.020130 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:55.023624 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:55.024283 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:55.024290 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:55.024295 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:55.024480 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:55.025011 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:55.025016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:55.025019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:55.025371 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:55.025511 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:55.026087 (+ 576us) server_negotiation.cc:299] Negotiation successful
0812 07:09:55.026425 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 07:09:55.027601 696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:55.010836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.61:40703 (local address 127.0.0.1:37272)
0812 07:09:55.011239 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:55.011266 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 07:09:55.011361 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:09:55.011948 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:09:55.011953 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:09:55.012410 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:09:55.012601 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:55.012612 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.014015 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.014021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:55.015275 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:55.015290 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.016318 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.016324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:55.016524 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:55.017102 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:09:55.017124 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:09:55.019891 (+ 2767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:09:55.023771 (+ 3880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:55.023776 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:55.023780 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:55.024157 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:55.024635 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:55.024643 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:55.024648 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:55.024838 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:55.025580 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:09:55.025585 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:09:55.025762 (+ 177us) client_negotiation.cc:769] Sending connection context
0812 07:09:55.026964 (+ 1202us) client_negotiation.cc:240] Negotiation successful
0812 07:09:55.027249 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":2}
I20260812 07:09:55.028748 689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.61.60" port: 36781 } has no permanent_uuid. Determining permanent_uuid...
I20260812 07:09:55.029857 695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:55.011740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42814 (local address 127.6.61.62:35063)
0812 07:09:55.011896 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:55.011902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:55.011918 (+ 16us) server_negotiation.cc:407] Connection header received
0812 07:09:55.011970 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:55.011975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:55.012057 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:55.012164 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:55.013800 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.014519 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.016256 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.016978 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.017101 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:55.021783 (+ 4682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:55.021788 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:55.021799 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:55.021883 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:55.025924 (+ 4041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:55.027202 (+ 1278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:55.027209 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:55.027213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:55.027366 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:55.027814 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:55.027822 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:55.027826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:55.028192 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:55.028511 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:55.029409 (+ 898us) server_negotiation.cc:299] Negotiation successful
0812 07:09:55.029594 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 07:09:55.032693 524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9889f2c00e9744098278434e2f800153: Bootstrap starting.
Aug 12 07:09:55 dist-test-slave-69rg krb5kdc[435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518594, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.60@KRBTEST.COM for kudu/127.6.61.60@KRBTEST.COM
I20260812 07:09:55.037026 524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9889f2c00e9744098278434e2f800153: Neither blocks nor log segments found. Creating new log.
I20260812 07:09:55.038519 524 log.cc:824] T 00000000000000000000000000000000 P 9889f2c00e9744098278434e2f800153: Log is configured to *not* fsync() on all Append() calls
Aug 12 07:09:55 dist-test-slave-69rg krb5kdc[435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.61.61@KRBTEST.COM
I20260812 07:09:55.042085 524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9889f2c00e9744098278434e2f800153: No bootstrap required, opened a new log
I20260812 07:09:55.044574 696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:55.029444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:36781 (local address 127.0.0.1:49340)
0812 07:09:55.029994 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:55.030011 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:09:55.030120 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:09:55.030541 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:09:55.030544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:09:55.031000 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:09:55.031169 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:55.031180 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.032290 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.032295 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:55.033530 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:55.033542 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.034321 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.034326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:55.034570 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:55.035153 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:09:55.035170 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:09:55.037514 (+ 2344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:09:55.041400 (+ 3886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:55.041407 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:55.041413 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:55.041946 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:55.042496 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:55.042502 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:55.042506 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:55.042677 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:55.043424 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:09:55.043432 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:09:55.043595 (+ 163us) client_negotiation.cc:769] Sending connection context
0812 07:09:55.043770 (+ 175us) client_negotiation.cc:240] Negotiation successful
0812 07:09:55.044119 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":1}
I20260812 07:09:55.045028 693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:55.029708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49340 (local address 127.6.61.60:36781)
0812 07:09:55.029888 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:55.029893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:55.030093 (+ 200us) server_negotiation.cc:407] Connection header received
0812 07:09:55.030227 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:55.030233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:55.030308 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:55.030402 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:55.031321 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.032158 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.033701 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.034183 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.034311 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:55.037694 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:55.037700 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:55.037704 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:55.037741 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:55.041250 (+ 3509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:55.042200 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:55.042205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:55.042208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:55.042377 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:55.042807 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:55.042815 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:55.042820 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:55.043261 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:55.043450 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:55.043901 (+ 451us) server_negotiation.cc:299] Negotiation successful
0812 07:09:55.044326 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":159,"server-negotiator.queue_time_us":45}
I20260812 07:09:55.049957 613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:55.031926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37278 (local address 127.6.61.61:40703)
0812 07:09:55.032079 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:55.032085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:55.032332 (+ 247us) server_negotiation.cc:407] Connection header received
0812 07:09:55.032502 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:55.032507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:55.032582 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:55.032680 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:55.034356 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.035510 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.037513 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.038068 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.038221 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:55.043208 (+ 4987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:55.043219 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:55.043222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:55.043261 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:55.046657 (+ 3396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:55.047399 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:55.047406 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:55.047409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:55.047554 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:55.048060 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:55.048066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:55.048070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:55.048384 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:55.048699 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:55.049146 (+ 447us) server_negotiation.cc:299] Negotiation successful
0812 07:09:55.049497 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 07:09:55.053422 689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0cd65fa649b462ca72e4f51dfcdf71c: Bootstrap starting.
I20260812 07:09:55.054066 524 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9889f2c00e9744098278434e2f800153 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9889f2c00e9744098278434e2f800153" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35063 } } peers { permanent_uuid: "006e588b5094425eac49d60068a86573" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 40703 } } peers { permanent_uuid: "c0cd65fa649b462ca72e4f51dfcdf71c" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 36781 } }
I20260812 07:09:55.054479 524 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9889f2c00e9744098278434e2f800153 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9889f2c00e9744098278434e2f800153, State: Initialized, Role: FOLLOWER
I20260812 07:09:55.055007 524 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9889f2c00e9744098278434e2f800153 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9889f2c00e9744098278434e2f800153" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35063 } } peers { permanent_uuid: "006e588b5094425eac49d60068a86573" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 40703 } } peers { permanent_uuid: "c0cd65fa649b462ca72e4f51dfcdf71c" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 36781 } }
I20260812 07:09:55.055656 6388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:09:55.058068 498 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9889f2c00e9744098278434e2f800153"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42814
I20260812 07:09:55.058444 689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0cd65fa649b462ca72e4f51dfcdf71c: Neither blocks nor log segments found. Creating new log.
I20260812 07:09:55.060171 689 log.cc:824] T 00000000000000000000000000000000 P c0cd65fa649b462ca72e4f51dfcdf71c: Log is configured to *not* fsync() on all Append() calls
I20260812 07:09:55.062407 698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9889f2c00e9744098278434e2f800153 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9889f2c00e9744098278434e2f800153" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35063 } } peers { permanent_uuid: "006e588b5094425eac49d60068a86573" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 40703 } } peers { permanent_uuid: "c0cd65fa649b462ca72e4f51dfcdf71c" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 36781 } } }
I20260812 07:09:55.062858 698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9889f2c00e9744098278434e2f800153 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:09:55.063807 524 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9889f2c00e9744098278434e2f800153 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:09:55.063903 689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0cd65fa649b462ca72e4f51dfcdf71c: No bootstrap required, opened a new log
I20260812 07:09:55.063879 498 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9889f2c00e9744098278434e2f800153 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 07:09:55.064121 498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9889f2c00e9744098278434e2f800153 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:09:55.067183 498 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9889f2c00e9744098278434e2f800153 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9889f2c00e9744098278434e2f800153" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35063 } } peers { permanent_uuid: "006e588b5094425eac49d60068a86573" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 40703 } } peers { permanent_uuid: "c0cd65fa649b462ca72e4f51dfcdf71c" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 36781 } }
I20260812 07:09:55.068635 498 leader_election.cc:288] T 00000000000000000000000000000000 P 9889f2c00e9744098278434e2f800153 [CANDIDATE]: Term 1 election: Requested vote from peers 006e588b5094425eac49d60068a86573 (127.6.61.61:40703), c0cd65fa649b462ca72e4f51dfcdf71c (127.6.61.60:36781)
I20260812 07:09:55.077790 689 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0cd65fa649b462ca72e4f51dfcdf71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9889f2c00e9744098278434e2f800153" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35063 } } peers { permanent_uuid: "006e588b5094425eac49d60068a86573" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 40703 } } peers { permanent_uuid: "c0cd65fa649b462ca72e4f51dfcdf71c" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 36781 } }
I20260812 07:09:55.078342 689 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0cd65fa649b462ca72e4f51dfcdf71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0cd65fa649b462ca72e4f51dfcdf71c, State: Initialized, Role: FOLLOWER
I20260812 07:09:55.078877 689 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0cd65fa649b462ca72e4f51dfcdf71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9889f2c00e9744098278434e2f800153" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35063 } } peers { permanent_uuid: "006e588b5094425eac49d60068a86573" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 40703 } } peers { permanent_uuid: "c0cd65fa649b462ca72e4f51dfcdf71c" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 36781 } }
I20260812 07:09:55.086030 716 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0cd65fa649b462ca72e4f51dfcdf71c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9889f2c00e9744098278434e2f800153" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35063 } } peers { permanent_uuid: "006e588b5094425eac49d60068a86573" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 40703 } } peers { permanent_uuid: "c0cd65fa649b462ca72e4f51dfcdf71c" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 36781 } } }
I20260812 07:09:55.086491 716 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0cd65fa649b462ca72e4f51dfcdf71c [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:09:55.088617 689 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0cd65fa649b462ca72e4f51dfcdf71c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:09:55.088670 660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9889f2c00e9744098278434e2f800153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c0cd65fa649b462ca72e4f51dfcdf71c"
I20260812 07:09:55.089260 660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0cd65fa649b462ca72e4f51dfcdf71c [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:09:55.093060 660 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c0cd65fa649b462ca72e4f51dfcdf71c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9889f2c00e9744098278434e2f800153 in term 1.
I20260812 07:09:55.093802 471 leader_election.cc:302] T 00000000000000000000000000000000 P 9889f2c00e9744098278434e2f800153 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9889f2c00e9744098278434e2f800153, c0cd65fa649b462ca72e4f51dfcdf71c; no voters:
I20260812 07:09:55.094431 698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9889f2c00e9744098278434e2f800153 [term 1 FOLLOWER]: Leader election won for term 1
W20260812 07:09:55.100162 475 tablet.cc:2369] T 00000000000000000000000000000000 P 9889f2c00e9744098278434e2f800153: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:09:55.106544 698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9889f2c00e9744098278434e2f800153 [term 1 LEADER]: Becoming Leader. State: Replica: 9889f2c00e9744098278434e2f800153, State: Running, Role: LEADER
I20260812 07:09:55.107440 698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9889f2c00e9744098278434e2f800153 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9889f2c00e9744098278434e2f800153" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35063 } } peers { permanent_uuid: "006e588b5094425eac49d60068a86573" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 40703 } } peers { permanent_uuid: "c0cd65fa649b462ca72e4f51dfcdf71c" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 36781 } }
I20260812 07:09:55.112425 695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:55.073791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42830 (local address 127.6.61.62:35063)
0812 07:09:55.074193 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:55.074201 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:55.074767 (+ 566us) server_negotiation.cc:407] Connection header received
0812 07:09:55.074987 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:55.074995 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:55.075095 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:55.075222 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:55.077889 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.079005 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.083968 (+ 4963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.084907 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.085043 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:55.094904 (+ 9861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:55.094914 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:55.094922 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:55.094978 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:55.099673 (+ 4695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:55.103038 (+ 3365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:55.103047 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:55.103090 (+ 43us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:55.103512 (+ 422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:55.105875 (+ 2363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:55.105915 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:55.105923 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:55.106913 (+ 990us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:55.107671 (+ 758us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:55.110647 (+ 2976us) server_negotiation.cc:299] Negotiation successful
0812 07:09:55.111349 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271}
I20260812 07:09:55.112982 613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:55.073647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37280 (local address 127.6.61.61:40703)
0812 07:09:55.074032 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:55.074042 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:55.077640 (+ 3598us) server_negotiation.cc:407] Connection header received
0812 07:09:55.078045 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:55.078054 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:55.078195 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:55.078294 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:55.083742 (+ 5448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.085193 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.093850 (+ 8657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.094536 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.094658 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:55.099352 (+ 4694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:55.099362 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:55.099370 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:55.099437 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:55.105129 (+ 5692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:55.109241 (+ 4112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:55.109254 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:55.109258 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:55.109446 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:55.110323 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:55.110328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:55.110331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:55.110972 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:55.111348 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:55.112223 (+ 875us) server_negotiation.cc:299] Negotiation successful
0812 07:09:55.112568 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232}
I20260812 07:09:55.122316 693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:55.075788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49350 (local address 127.6.61.60:36781)
0812 07:09:55.075975 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:55.075982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:55.087200 (+ 11218us) server_negotiation.cc:407] Connection header received
0812 07:09:55.087472 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:55.087479 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:55.087595 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:55.087721 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:55.090298 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.091386 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.100860 (+ 9474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.101656 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.101785 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:55.113584 (+ 11799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:55.113595 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:55.113603 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:55.113660 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:55.117725 (+ 4065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:55.119051 (+ 1326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:55.119060 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:55.119066 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:55.119266 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:55.119851 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:55.119859 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:55.119865 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:55.120339 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:55.120695 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:55.121459 (+ 764us) server_negotiation.cc:299] Negotiation successful
0812 07:09:55.121781 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 07:09:55.125411 719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9889f2c00e9744098278434e2f800153 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9889f2c00e9744098278434e2f800153. Latest consensus state: current_term: 1 leader_uuid: "9889f2c00e9744098278434e2f800153" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9889f2c00e9744098278434e2f800153" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35063 } } peers { permanent_uuid: "006e588b5094425eac49d60068a86573" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 40703 } } peers { permanent_uuid: "c0cd65fa649b462ca72e4f51dfcdf71c" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 36781 } } }
I20260812 07:09:55.126006 719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9889f2c00e9744098278434e2f800153 [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:55.127523 733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:09:55.137887 733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:09:55.152134 606 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.61.60" port: 36781 } attempt: 5
I20260812 07:09:55.158371 450 master_runner.cc:186] Error getting master registration for 127.6.61.61:40703: OK, instance_id { permanent_uuid: "006e588b5094425eac49d60068a86573" instance_seqno: 1786518594417069 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 07:09:55.169378 450 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 12 07:09:55 dist-test-slave-69rg krb5kdc[435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518594, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.61.61@KRBTEST.COM for kudu/127.6.61.60@KRBTEST.COM
I20260812 07:09:55.167484 616 master_runner.cc:186] Error getting master registration for 127.6.61.61:40703: OK, instance_id { permanent_uuid: "006e588b5094425eac49d60068a86573" instance_seqno: 1786518594417069 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 07:09:55.177686 616 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 07:09:55.182091 660 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c0cd65fa649b462ca72e4f51dfcdf71c [term 1 FOLLOWER]: Refusing update from remote peer 9889f2c00e9744098278434e2f800153: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:09:55.182829 693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:55.153444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49360 (local address 127.6.61.60:36781)
0812 07:09:55.153618 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:55.153624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:55.153642 (+ 18us) server_negotiation.cc:407] Connection header received
0812 07:09:55.153710 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:55.153716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:55.153795 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:55.153924 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:55.155499 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.156591 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.171330 (+ 14739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.172090 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.172247 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:55.176333 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:09:55.176341 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:09:55.176351 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:09:55.176396 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:09:55.179635 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:55.180487 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:55.180495 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:55.180498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:55.180653 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:09:55.181107 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:09:55.181112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:09:55.181115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:09:55.181504 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:09:55.181706 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:55.182189 (+ 483us) server_negotiation.cc:299] Negotiation successful
0812 07:09:55.182445 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 07:09:55.183120 719 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9889f2c00e9744098278434e2f800153 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0cd65fa649b462ca72e4f51dfcdf71c" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 36781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:09:55.184890 610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:55.152721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.61.60:36781 (local address 127.0.0.1:49360)
0812 07:09:55.153092 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0812 07:09:55.153119 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 07:09:55.153240 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:09:55.154512 (+ 1272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:09:55.154517 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:09:55.155092 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:09:55.155312 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:55.155325 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.169544 (+ 14219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.169551 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:55.171119 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:09:55.171137 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.172254 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.172260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:09:55.172492 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:55.173260 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:09:55.173287 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:09:55.176083 (+ 2796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:09:55.179796 (+ 3713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:55.179802 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:55.179806 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:55.180326 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:55.180768 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:09:55.180774 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:09:55.180777 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:09:55.180967 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:09:55.181667 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:09:55.181672 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:09:55.181931 (+ 259us) client_negotiation.cc:769] Sending connection context
0812 07:09:55.182091 (+ 160us) client_negotiation.cc:240] Negotiation successful
0812 07:09:55.182420 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241}
I20260812 07:09:55.190999 606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 006e588b5094425eac49d60068a86573: Bootstrap starting.
W20260812 07:09:55.191406 740 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c0cd65fa649b462ca72e4f51dfcdf71c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 07:09:55.191574 740 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 07:09:55.195329 606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 006e588b5094425eac49d60068a86573: Neither blocks nor log segments found. Creating new log.
I20260812 07:09:55.196695 606 log.cc:824] T 00000000000000000000000000000000 P 006e588b5094425eac49d60068a86573: Log is configured to *not* fsync() on all Append() calls
I20260812 07:09:55.203210 606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 006e588b5094425eac49d60068a86573: No bootstrap required, opened a new log
W20260812 07:09:55.205719 638 tablet.cc:2369] T 00000000000000000000000000000000 P c0cd65fa649b462ca72e4f51dfcdf71c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:09:55.212114 716 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0cd65fa649b462ca72e4f51dfcdf71c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9889f2c00e9744098278434e2f800153. Latest consensus state: current_term: 1 leader_uuid: "9889f2c00e9744098278434e2f800153" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9889f2c00e9744098278434e2f800153" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35063 } } peers { permanent_uuid: "006e588b5094425eac49d60068a86573" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 40703 } } peers { permanent_uuid: "c0cd65fa649b462ca72e4f51dfcdf71c" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 36781 } } }
I20260812 07:09:55.212466 716 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0cd65fa649b462ca72e4f51dfcdf71c [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:09:55.214957 606 raft_consensus.cc:348] T 00000000000000000000000000000000 P 006e588b5094425eac49d60068a86573 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9889f2c00e9744098278434e2f800153" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35063 } } peers { permanent_uuid: "006e588b5094425eac49d60068a86573" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 40703 } } peers { permanent_uuid: "c0cd65fa649b462ca72e4f51dfcdf71c" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 36781 } }
I20260812 07:09:55.215404 606 raft_consensus.cc:729] T 00000000000000000000000000000000 P 006e588b5094425eac49d60068a86573 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 006e588b5094425eac49d60068a86573, State: Initialized, Role: FOLLOWER
I20260812 07:09:55.215940 606 consensus_queue.cc:260] T 00000000000000000000000000000000 P 006e588b5094425eac49d60068a86573 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9889f2c00e9744098278434e2f800153" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35063 } } peers { permanent_uuid: "006e588b5094425eac49d60068a86573" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 40703 } } peers { permanent_uuid: "c0cd65fa649b462ca72e4f51dfcdf71c" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 36781 } }
I20260812 07:09:55.225246 742 mvcc.cc:205] Tried to move back new op lower bound from 7317580165854711808 to 7317580165567533056. Current Snapshot: MvccSnapshot[applied={T|T < 7317580165854711808}]
I20260812 07:09:55.226150 745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 006e588b5094425eac49d60068a86573 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9889f2c00e9744098278434e2f800153" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35063 } } peers { permanent_uuid: "006e588b5094425eac49d60068a86573" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 40703 } } peers { permanent_uuid: "c0cd65fa649b462ca72e4f51dfcdf71c" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 36781 } } }
I20260812 07:09:55.227380 745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 006e588b5094425eac49d60068a86573 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:09:55.229477 719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9889f2c00e9744098278434e2f800153 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9889f2c00e9744098278434e2f800153" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9889f2c00e9744098278434e2f800153" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35063 } } peers { permanent_uuid: "006e588b5094425eac49d60068a86573" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 40703 } } peers { permanent_uuid: "c0cd65fa649b462ca72e4f51dfcdf71c" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 36781 } } }
I20260812 07:09:55.229902 719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9889f2c00e9744098278434e2f800153 [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:55.231083 606 sys_catalog.cc:564] T 00000000000000000000000000000000 P 006e588b5094425eac49d60068a86573 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:09:55.231524 580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 006e588b5094425eac49d60068a86573 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:09:55.233932 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9889f2c00e9744098278434e2f800153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "006e588b5094425eac49d60068a86573"
I20260812 07:09:55.236959 580 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 006e588b5094425eac49d60068a86573 [term 1 FOLLOWER]: Refusing update from remote peer 9889f2c00e9744098278434e2f800153: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:09:55.240793 581 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 006e588b5094425eac49d60068a86573 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9889f2c00e9744098278434e2f800153 in term 1.
I20260812 07:09:55.242962 698 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9889f2c00e9744098278434e2f800153 [LEADER]: Connected to new peer: Peer: permanent_uuid: "006e588b5094425eac49d60068a86573" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 40703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:09:55.246914 716 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0cd65fa649b462ca72e4f51dfcdf71c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9889f2c00e9744098278434e2f800153" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9889f2c00e9744098278434e2f800153" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35063 } } peers { permanent_uuid: "006e588b5094425eac49d60068a86573" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 40703 } } peers { permanent_uuid: "c0cd65fa649b462ca72e4f51dfcdf71c" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 36781 } } }
I20260812 07:09:55.247176 716 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0cd65fa649b462ca72e4f51dfcdf71c [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:09:55.248781 733 catalog_manager.cc:1347] Generated new cluster ID: a8dee0d6c076428fb84badd46c2150f4
I20260812 07:09:55.248925 733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:09:55.270118 733 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:09:55.273845 733 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:09:55.283174 752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9889f2c00e9744098278434e2f800153 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9889f2c00e9744098278434e2f800153" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9889f2c00e9744098278434e2f800153" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35063 } } peers { permanent_uuid: "006e588b5094425eac49d60068a86573" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 40703 } } peers { permanent_uuid: "c0cd65fa649b462ca72e4f51dfcdf71c" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 36781 } } }
I20260812 07:09:55.283661 752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9889f2c00e9744098278434e2f800153 [sys.catalog]: This master's current role is: LEADER
I20260812 07:09:55.284934 745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 006e588b5094425eac49d60068a86573 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9889f2c00e9744098278434e2f800153. Latest consensus state: current_term: 1 leader_uuid: "9889f2c00e9744098278434e2f800153" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9889f2c00e9744098278434e2f800153" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35063 } } peers { permanent_uuid: "006e588b5094425eac49d60068a86573" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 40703 } } peers { permanent_uuid: "c0cd65fa649b462ca72e4f51dfcdf71c" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 36781 } } }
I20260812 07:09:55.285271 745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 006e588b5094425eac49d60068a86573 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:09:55.285581 756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 006e588b5094425eac49d60068a86573 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9889f2c00e9744098278434e2f800153" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9889f2c00e9744098278434e2f800153" member_type: VOTER last_known_addr { host: "127.6.61.62" port: 35063 } } peers { permanent_uuid: "006e588b5094425eac49d60068a86573" member_type: VOTER last_known_addr { host: "127.6.61.61" port: 40703 } } peers { permanent_uuid: "c0cd65fa649b462ca72e4f51dfcdf71c" member_type: VOTER last_known_addr { host: "127.6.61.60" port: 36781 } } }
I20260812 07:09:55.286056 756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 006e588b5094425eac49d60068a86573 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:09:55.303153 733 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9889f2c00e9744098278434e2f800153: Generated new TSK 0
I20260812 07:09:55.310987 733 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:09:55.335978 767 catalog_manager.cc:1259] Loaded cluster ID: a8dee0d6c076428fb84badd46c2150f4
I20260812 07:09:55.336194 767 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 006e588b5094425eac49d60068a86573: loading cluster ID for follower catalog manager: success
I20260812 07:09:55.340476 767 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 006e588b5094425eac49d60068a86573: acquiring CA information for follower catalog manager: success
I20260812 07:09:55.343513 767 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 006e588b5094425eac49d60068a86573: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 07:09:55.477180 613 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:09:55.464157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37284 (local address 127.6.61.61:40703)
0812 07:09:55.464333 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:55.464339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:55.474579 (+ 10240us) server_negotiation.cc:407] Connection header received
0812 07:09:55.474982 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:55.474992 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:55.475147 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:55.475245 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:55.476848 (+ 1603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37284: BlockingRecv error: recv got EOF from 127.0.0.1:37284 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260812 07:09:55.479840 693 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:09:55.471254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49370 (local address 127.6.61.60:36781)
0812 07:09:55.471482 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:55.471489 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:55.477998 (+ 6509us) server_negotiation.cc:407] Connection header received
0812 07:09:55.478323 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:55.478329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:55.478434 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:55.478517 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:55.479582 (+ 1065us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49370: BlockingRecv error: recv got EOF from 127.0.0.1:49370 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260812 07:09:55.480062 695 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:09:55.463490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42834 (local address 127.6.61.62:35063)
0812 07:09:55.464006 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:55.464014 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:55.477909 (+ 13895us) server_negotiation.cc:407] Connection header received
0812 07:09:55.478635 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:55.478644 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:55.478736 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:55.478811 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:55.479804 (+ 993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42834: BlockingRecv error: recv got EOF from 127.0.0.1:42834 (error 108)
Metrics: {"server-negotiator.queue_time_us":330}
I20260812 07:09:55.545689 613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:55.526164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37286 (local address 127.6.61.61:40703)
0812 07:09:55.526374 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:55.526381 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:55.537118 (+ 10737us) server_negotiation.cc:407] Connection header received
0812 07:09:55.537509 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:55.537516 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:55.537787 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:55.537984 (+ 197us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:09:55.538764 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.539957 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.542037 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.542761 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.542863 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:55.544036 (+ 1173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:09:55.544775 (+ 739us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:09:55.544894 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:55.545171 (+ 277us) server_negotiation.cc:299] Negotiation successful
0812 07:09:55.545284 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260812 07:09:55.592928 693 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:09:55.582719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49380 (local address 127.6.61.60:36781)
0812 07:09:55.582972 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:55.582978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:55.590582 (+ 7604us) server_negotiation.cc:407] Connection header received
0812 07:09:55.591212 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:55.591219 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:55.591351 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:55.591434 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:55.592663 (+ 1229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49380: BlockingRecv error: recv got EOF from 127.0.0.1:49380 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 07:09:55.643512 613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:55.625095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37300 (local address 127.6.61.61:40703)
0812 07:09:55.625291 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:55.625298 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:55.636236 (+ 10938us) server_negotiation.cc:407] Connection header received
0812 07:09:55.636879 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:55.636886 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:55.637014 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:55.637091 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:09:55.637877 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.639108 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.641114 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.641801 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.641946 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:55.642588 (+ 642us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:09:55.642770 (+ 182us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:09:55.642836 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:55.643058 (+ 222us) server_negotiation.cc:299] Negotiation successful
0812 07:09:55.643152 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260812 07:09:55.687507 693 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:09:55.677527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49390 (local address 127.6.61.60:36781)
0812 07:09:55.677740 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:55.677748 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:55.685295 (+ 7547us) server_negotiation.cc:407] Connection header received
0812 07:09:55.685914 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:55.685922 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:55.686028 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:55.686116 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:55.687225 (+ 1109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49390: BlockingRecv error: recv got EOF from 127.0.0.1:49390 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 07:09:55.738634 613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:55.723334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37304 (local address 127.6.61.61:40703)
0812 07:09:55.723527 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:55.723534 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:55.730467 (+ 6933us) server_negotiation.cc:407] Connection header received
0812 07:09:55.730858 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:55.730864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:55.730971 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:55.731055 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:09:55.732339 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.733599 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.735963 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.736715 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.736835 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:55.737573 (+ 738us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:09:55.737743 (+ 170us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:09:55.737840 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:55.738132 (+ 292us) server_negotiation.cc:299] Negotiation successful
0812 07:09:55.738252 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260812 07:09:55.785611 693 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:09:55.775288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49394 (local address 127.6.61.60:36781)
0812 07:09:55.775528 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:55.775536 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:55.783386 (+ 7850us) server_negotiation.cc:407] Connection header received
0812 07:09:55.784006 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:55.784014 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:55.784126 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:55.784221 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:55.785299 (+ 1078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49394: BlockingRecv error: recv got EOF from 127.0.0.1:49394 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
I20260812 07:09:55.845289 613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:55.826925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37314 (local address 127.6.61.61:40703)
0812 07:09:55.827116 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:55.827123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:55.838360 (+ 11237us) server_negotiation.cc:407] Connection header received
0812 07:09:55.838688 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:55.838697 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:55.838794 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:55.838863 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:09:55.839591 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.840784 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.842814 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.843506 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.843617 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:55.844342 (+ 725us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:09:55.844506 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:09:55.844567 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:55.844806 (+ 239us) server_negotiation.cc:299] Negotiation successful
0812 07:09:55.844901 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260812 07:09:55.888947 693 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:09:55.878861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49402 (local address 127.6.61.60:36781)
0812 07:09:55.879126 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:55.879133 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:55.886672 (+ 7539us) server_negotiation.cc:407] Connection header received
0812 07:09:55.887214 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:55.887221 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:55.887344 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:55.887425 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:55.888694 (+ 1269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49402: BlockingRecv error: recv got EOF from 127.0.0.1:49402 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
I20260812 07:09:55.920848 613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:55.914837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37326 (local address 127.6.61.61:40703)
0812 07:09:55.915100 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:55.915115 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:55.915734 (+ 619us) server_negotiation.cc:407] Connection header received
0812 07:09:55.915975 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:55.915983 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:55.916062 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:55.916175 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:09:55.916647 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.917400 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.918970 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.919425 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.919590 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:55.920020 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:09:55.920130 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:09:55.920235 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:55.920394 (+ 159us) server_negotiation.cc:299] Negotiation successful
0812 07:09:55.920482 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260812 07:09:55.935091 693 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:09:55.932674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49412 (local address 127.6.61.60:36781)
0812 07:09:55.932875 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:55.932883 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:55.933454 (+ 571us) server_negotiation.cc:407] Connection header received
0812 07:09:55.933709 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:55.933715 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:55.933804 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:55.933956 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:55.934774 (+ 818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49412: BlockingRecv error: recv got EOF from 127.0.0.1:49412 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 07:09:55.959609 613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:55.954209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37338 (local address 127.6.61.61:40703)
0812 07:09:55.954406 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:55.954411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:55.954787 (+ 376us) server_negotiation.cc:407] Connection header received
0812 07:09:55.954914 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:55.954918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:55.954990 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:55.955080 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:09:55.955592 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.956396 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.957863 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:55.958411 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:55.958524 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:55.958936 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:09:55.959040 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:09:55.959188 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:55.959336 (+ 148us) server_negotiation.cc:299] Negotiation successful
0812 07:09:55.959415 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260812 07:09:55.967576 693 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:09:55.965817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49420 (local address 127.6.61.60:36781)
0812 07:09:55.966053 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:55.966060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:55.966287 (+ 227us) server_negotiation.cc:407] Connection header received
0812 07:09:55.966420 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:55.966424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:55.966498 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:55.966582 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:55.967353 (+ 771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49420: BlockingRecv error: recv got EOF from 127.0.0.1:49420 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 07:09:56.009034 613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:56.003444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37344 (local address 127.6.61.61:40703)
0812 07:09:56.003708 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:56.003714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:56.003993 (+ 279us) server_negotiation.cc:407] Connection header received
0812 07:09:56.004297 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:56.004302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:56.004387 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:56.004500 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:09:56.004969 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:56.005696 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:56.007120 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:56.007591 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:56.007765 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:56.008198 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:09:56.008331 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:09:56.008441 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:56.008605 (+ 164us) server_negotiation.cc:299] Negotiation successful
0812 07:09:56.008697 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260812 07:09:56.017951 693 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:09:56.015784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49424 (local address 127.6.61.60:36781)
0812 07:09:56.016082 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:56.016091 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:56.016420 (+ 329us) server_negotiation.cc:407] Connection header received
0812 07:09:56.016662 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:56.016667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:56.016763 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:56.016878 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:56.017690 (+ 812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49424: BlockingRecv error: recv got EOF from 127.0.0.1:49424 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
I20260812 07:09:56.091820 613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:56.086061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37350 (local address 127.6.61.61:40703)
0812 07:09:56.086348 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:56.086356 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:56.086642 (+ 286us) server_negotiation.cc:407] Connection header received
0812 07:09:56.086860 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:56.086865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:56.086947 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:56.087041 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:09:56.087505 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:56.088293 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:56.089769 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:56.090392 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:56.090502 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:56.090959 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:09:56.091080 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:09:56.091187 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:56.091407 (+ 220us) server_negotiation.cc:299] Negotiation successful
0812 07:09:56.091514 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260812 07:09:56.100059 693 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:09:56.098058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49436 (local address 127.6.61.60:36781)
0812 07:09:56.098349 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:56.098358 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:56.098656 (+ 298us) server_negotiation.cc:407] Connection header received
0812 07:09:56.098878 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:56.098883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:56.098958 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:56.099067 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:09:56.099890 (+ 823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49436: BlockingRecv error: recv got EOF from 127.0.0.1:49436 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
I20260812 07:09:56.193713 740 catalog_manager.cc:1259] Loaded cluster ID: a8dee0d6c076428fb84badd46c2150f4
I20260812 07:09:56.193892 740 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c0cd65fa649b462ca72e4f51dfcdf71c: loading cluster ID for follower catalog manager: success
I20260812 07:09:56.196791 740 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c0cd65fa649b462ca72e4f51dfcdf71c: acquiring CA information for follower catalog manager: success
I20260812 07:09:56.198616 740 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c0cd65fa649b462ca72e4f51dfcdf71c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 07:09:56.237933 613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:56.232317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37356 (local address 127.6.61.61:40703)
0812 07:09:56.232599 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:56.232608 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:56.232899 (+ 291us) server_negotiation.cc:407] Connection header received
0812 07:09:56.233188 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:56.233211 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:56.233347 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:56.233465 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:09:56.233947 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:56.234664 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:56.236047 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:56.236582 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:56.236729 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:56.237143 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:09:56.237252 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:09:56.237352 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:56.237515 (+ 163us) server_negotiation.cc:299] Negotiation successful
0812 07:09:56.237598 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260812 07:09:56.250576 693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:56.244230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49448 (local address 127.6.61.60:36781)
0812 07:09:56.244502 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:56.244507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:56.244600 (+ 93us) server_negotiation.cc:407] Connection header received
0812 07:09:56.244827 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:56.244832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:56.245007 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:56.245115 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:09:56.245584 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:56.246383 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:56.247895 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:56.248387 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:56.248492 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:56.249084 (+ 592us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:09:56.249654 (+ 570us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:09:56.249771 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:56.249992 (+ 221us) server_negotiation.cc:299] Negotiation successful
0812 07:09:56.250077 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260812 07:09:56.274636 885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:56.258156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42848 (local address 127.6.61.62:35063)
0812 07:09:56.266881 (+ 8725us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:56.266891 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:56.266919 (+ 28us) server_negotiation.cc:407] Connection header received
0812 07:09:56.267083 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:56.267091 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:56.267400 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:56.267657 (+ 257us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:09:56.268489 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:56.269663 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:56.271873 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:56.272276 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:56.272423 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:56.273049 (+ 626us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:09:56.273515 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:09:56.273630 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:56.274197 (+ 567us) server_negotiation.cc:299] Negotiation successful
0812 07:09:56.274292 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8567,"thread_start_us":8411,"threads_started":1}
I20260812 07:09:56.275157 613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:56.258209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37364 (local address 127.6.61.61:40703)
0812 07:09:56.258804 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:56.258813 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:56.268645 (+ 9832us) server_negotiation.cc:407] Connection header received
0812 07:09:56.268861 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:56.268867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:56.268958 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:56.269053 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:09:56.269928 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:56.270918 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:56.272981 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:56.273593 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:56.273726 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:56.274321 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:09:56.274449 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:09:56.274538 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:56.274768 (+ 230us) server_negotiation.cc:299] Negotiation successful
0812 07:09:56.274858 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353}
I20260812 07:09:56.275859 693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:09:56.258474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49452 (local address 127.6.61.60:36781)
0812 07:09:56.259007 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0812 07:09:56.259016 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:09:56.269390 (+ 10374us) server_negotiation.cc:407] Connection header received
0812 07:09:56.269630 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:09:56.269640 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:09:56.269712 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:09:56.269824 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:09:56.270553 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:56.271651 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:56.273606 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:09:56.274363 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:09:56.274504 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:09:56.275042 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:09:56.275177 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:09:56.275273 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 07:09:56.275521 (+ 248us) server_negotiation.cc:299] Negotiation successful
0812 07:09:56.275610 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363}
I20260812 07:09:56.280309 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 450
I20260812 07:09:56.281802 519 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:09:56.399443 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 450
I20260812 07:09:56.428121 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 533
W20260812 07:09:56.428128 553 connection.cc:535] server connection from 127.0.0.1:37266 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 07:09:56.429359 601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:09:56.540556 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 533
I20260812 07:09:56.563361 6388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6k_OM2/build/asan/bin/kudu pid 616
I20260812 07:09:56.564584 684 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 07:09:56.674808 6388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6k_OM2/build/asan/bin/kudu with pid 616
2026-08-12T07:09:56Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4892 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10369 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (387959 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST